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ABSTRACT 


This  thesis  discusses  risk  in  Department  of  Defense 
(DoD)  weapon  systems  acquisition.  It  uses  the  Marine 
Corps'  Advanced  Amphibious  Assault  Vehicle  (AAAV)  as  a  case 
study  in  risk  management  strategy  and  techniques. 

The  AAAV  will  provide  the  Marine  Corps  with  a  fast 
deploying,  over-the-horizon,  and  waterborne  insertion 
capability.  The  AAAV' s  improvements  over  the  currently 
fielded  Amphibious  Assault  Vehicle  (AAV)  will  provide 
Marines  with  a  highly  survivable  and  lethal  weapon  system 
ashore . 

Risk  is  the  possibility  of  damage,  injury  or  loss. 
The  severity  of  a  risk  is  determined  by  a  combination  of 
both  the  probability  of  an  unfavorable  event  occurring  and 
the  severity  of  the  event's  occurrence. 

Risks  are  present  in  virtually  all  DoD  developmental 
programs.  Programs  suffer  from  risks  in  technical 
challenges,  unstable  system  requirements,  missing  schedule 
milestones,  unpredictable  funding  and  cost  overruns. 

The  DoD  currently  uses  techniques  to  mitigate  risks 
inherent  in  advanced  system  development.  This  thesis 
analyzes  the  AAAV' s  Program  Definition  and  Risk  Reduction 
(PDRR)  acquisition  phase  risk  management  strategy.  The 
thesis  concludes  by  drawing  from  the  lessons  learned  in  the 
AAAV  program  during  PDRR  and  analyzing  the  application  of 
the  lessons  learned  during  the  AAAV' s  current  acquisition 
phase.  System  Development  and  Demonstration  (SDD) . 
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INTRODUCTION 


I . 


A.  PURPOSE 

This  thesis  examines  the  Department  of  Defense  (DoD) 
system  acquisition  risk  management  environment  by  analyzing 
the  Marine  Corps'  Advanced  Amphibious  Assault  Vehicle 
(AAAV)  program.  To  conduct  this  analysis,  this  thesis  will 
discuss  risk  in  the  context  of  DoD  development  and 
procurement,  current  risk  management  practices  in  DoD  and 
in  the  defense  industry,  and  introduce  the  AAAV  system  to 
briefly  familiarize  the  reader  with  the  program  .  The 

analysis  will  concentrate  on  the  AAAV  Program  Definition 
and  Risk  Reduction  (PDRR)  Acquisition  phase.  This  thesis 
will  discuss  the  AAAV' s  System  Development  and 

Demonstration  (SDD)  ,  the  current  Acquisition  Phase,  risk 
management  strategy  with  respect  to  lessons  learned  during 
PDRR.  This  thesis  will  conclude  by  examining  the  AAAV' s 
SDD  risk  management  practices  and  providing  recommendations 
for  managing  risk  in  developmental  weapons  system 

acquisition  based  on  the  AAAV' s  experiences. 

B .  BACKGROUND 

Risk  is  the  possibility  of  injury,  damage  or  loss.  In 
DoD  systems  acquisition,  risks  are  the  "chances  of  not 
achieving  the  results  as  planned."  (Forsberg,  Mooz  and 
Cotterman,  2000,  p.  188)  In  weapon  system  development  and 
procurement,  planned  results  are  meeting  operational 
deficiencies  throughout  DoD  on  time,  on  budget  and  to  a 
satisfactory  performance  level.  The  failure  to  satisfy  the 
war  fighter' s  requirements  can  result  in  decreased 
effectiveness  of  the  United  States  DoD. 
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Risk  is  the  "probability  or  likelihood  of  failing  to 
achieve  a  particular  outcome"  and  "the  consequence  or 
impact  of  failing  to  achieve  that  outcome."  (Defense 
Acquisition  University  (DAU) ,  Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  5)  A  level  of  risk  is  determined  by 
combining  both  the  probability  of  the  undesirable  event 
occurring  and  the  impact,  or  severity,  of  the  event.  There 
are  many  categories  of  risk.  This  thesis  discusses 

technical  risk,  requirements  risk,  schedule  risk  and 
cost/funding  risk. 

The  DoD  acquisition  regulations  are  undergoing  change 
at  the  time  this  thesis  is  being  written.  The  AAAV  program 
executed  its  risk  management  strategy  based  on  then  current 
DoD  acquisition  guidelines  and  regulations.  This  thesis 
discusses  risk  management  practices  designed  to  reduce, 
eliminate,  transfer  and  accept  risk  in  developmental 

programs.  The  purpose  of  this  research  and  analysis  is  to 
present  the  risk  management  techniques  the  AAAV  program  has 
benefited  from  most.  Additionally,  this  thesis  will 
discuss  which  aspects  of  the  program' s  PDRR  risk  management 
strategy  have  led  to  the  adoption  of  different  techniques 
in  SDD  and  discuss  why.  The  overall  benefit  of  this 
research  is  to  familiarize  the  reader  with  successful  risk 
management  practices  in  DoD  acquisition. 

C.  RESEARCH  QUESTIONS 

The  primary  research  question  this  thesis  addresses 
is : 

•  How  have  the  lessons  learned  from  the  AAAV' s 
Program  Definition  and  Risk  Reduction  (PDRR)  Risk 
Management  Strategy  impacted  the  Program' s  Risk 
Management  Process  during  System  Development  and 
Demonstration  (SDD) ? 
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In  order  to  answer  the  primary  research  question,  this 
thesis  will  answer  the  following  subsidiary  questions  to 
provide  the  necessary  background  information: 

•  What  are  risk  and  risk  management  in  Department 
of  Defense  (DoD)  systems  acquisition? 

•  What  techniques  can  DoD  use  to  manage  risk  in 
developmental  systems? 

•  What  is  the  AAAV  program? 

•  What  are  the  lessons  learned  from  the  AAAV  PDRR 
Risk  Management  Strategy? 

•  What  risk  management  approaches  has  the  AAAV 
Program  Office  adopted  to  manage  technical  and 
programmatic  risk  during  SDD? 

•  What  conclusions  and  recommendations  can  be  drawn 
from  this  analysis? 

D.  RESEARCH  METHODOLOGY 

This  author' s  research  methodology  included  extensive 
literary  and  Internet  searches.  The  primary  forms  of 
literature  used  were  DoD  publications  and  guidelines, 
magazine  articles  and  textbooks.  The  Internet  provided  a 
great  deal  of  information  on  DoD  risk  management  techniques 
and  on  the  AAAV.  Of  greatest  benefit  to  the  research  was 
the  opportunity  to  visit  the  AAAV  program  office  in 
Virginia.  This  author  was  able  to  interview  Government 
program  office  as  well  as  Prime  Contractor  personnel.  The 
information  and  insights  were  invaluable  to  this  effort. 

E.  ORGANIZATION  OF  THE  STUDY 

This  thesis  is  organized  into  six  chapters.  A  brief 
description  of  the  chapters'  content  follows. 

Chapter  I  introduces  the  thesis  and  the  primary  and 
subsidiary  thesis  questions.  The  purpose  of  this  chapter 
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is  to  provide  a  snapshot  of  the  thesis  and  its  intended 
benefit  to  readers. 

Chapter  II  provides  background  information  on  the  DoD 
risk  management  environment.  The  chapter  offers  the  reader 
the  information  necessary  to  better  appreciate  subsequent 
chapters.  Chapter  II  discusses  types  of  risk  commonly 
encountered  in  defense  acquisitions  and  presents  risk 
management  techniques  used  in  weapon  system  procurement  and 
development . 

Chapter  III  provides  the  reader  with  background 
information  on  the  AAAV  system  and  its  acquisition  history 
to  date.  The  purpose  of  Chapter  III  is  to  familiarize  the 
reader  with  the  challenges  and  complexities  of  developing  a 
system  like  the  AAAV. 

Chapter  IV  discusses  the  AAAV  PDRR  risk  management 
techniques.  This  chapter  presents  the  data  to  be  analyzed. 
The  chapter  will  focus  on  five  areas  of  risk  management  in 
the  AAAV  program  during  PDRR: 

•  Information  Technology  Tools 

•  Risk  Management  Process 

•  Managing  Risk  Through  the  Contracting  Process 

•  Government  and  Prime  Contractor  Co -location 

•  Test  and  Evaluation 

Chapter  V  analyzes  the  AAAV  PDRR  risk  management 
strategy  and  introduces  elements  of  the  AAAV  SDD  risk 
management  plan  based  on  PDRR  lessons  learned. 

Chapter  VI  concludes  the  thesis  by  summarizing  how  the 
lessons  learned  from  the  AAAV' s  PDRR  risk  management 
strategy  have  helped  shape  the  program's  current  risk 
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management  practices  in  SDD .  The  thesis  closes  by 
presenting  recommendations  for  managing  risk  in  DoD 
acquisition  programs  and  offering  areas  for  further 
research  and  study  in  DoD  acquisition  risk  management. 

F.  SUMMARY 

The  purpose  of  this  chapter  is  to  provide  the  reader 
with  an  overview  of  this  thesis.  The  benefit  of  this 
research  and  analysis  is  to  highlight  successful  risk 
management  techniques  in  complex,  developmental  weapon 
systems .  The  techniques  and  procedures  may  have 
application  to  managing  risk  in  any  program  or 
organization . 

The  next  chapter  provides  background  information  on 
the  DoD  risk  management  environment. 
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II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  defines  risk  in  the  context  of  Department 
of  Defense  (DoD)  program  management  and  systems 
acquisition.  The  chapter  then  analyzes  the  risk  management 
and  risk  mitigation  processes  in  DoD.  It  addresses  the 
importance  of  striking  a  balance  between  risk  acceptance 
and  risk  mitigation  in  a  developmental  weapon  system 
program.  This  chapter  concludes  by  exploring  different 
risk  management  techniques  commonly  used  throughout  the  DoD 
acquisition  environment. 

B.  RISK  IN  THE  DEPARTMENT  OF  DEFENSE 

Risk  is  the  possibility  of  injury,  damage  or  loss.  In 
program  management,  risks  are  the  "chances  of  not  achieving 
the  results  as  planned."  (Forsberg,  Mooz  and  Cotterman, 
2000,  p.  188)  With  rapid  technological  growth  and 
emerging,  complex  mission  needs,  risk  exists  in  virtually 
all  of  today's  DoD  developmental  weapons  systems.  In 
Defense  Acquisitions,  loss  refers  to  the  impact  of  the  risk 
to  a  program,  which  could  be  in  the  form  of  diminished 
performance,  increased  costs  or  schedule  delays.  Risk  is 
the  "probability  or  likelihood  of  failing  to  achieve  a 
particular  outcome"  and  "the  consequence  or  impact  of 
failing  to  achieve  that  outcome."  (Defense  Acquisition 
University  (DAU) ,  Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  5) 

Risk,  whether  programmatic,  technical,  managerial, 
etc.,  is  present  in  DoD  developmental  systems.  Numerous 
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risk  areas  exist  in  the  acquisition  environment,  each 
posing  a  threat  to  the  success  of  a  program. 

1.  Types  of  Risk 

Risks  are  future  events  that  may  or  may  not  occur.  In 
DoD  acquisitions,  risks  are  future  events  that  may 
adversely  affect  a  program's  cost  constraints,  schedule  or 
performance  requirements.  The  types  of  risk  are  often 
interrelated  and  are  not  always  obvious. 

Risks  are  in  the  Program  Management  Office  (PMO) 
(program  plans,  etc.);  in  support  provided  by 
other  Government  agencies;  in  threat  assessments; 
and  in  prime  contractor  processes,  engineering 
and  manufacturing  processes,  and  technology. 

(Risk  Management  Guide  for  DoD  Acquisition,  2001, 
p.  6-7) 

A  Program  Manager  (PM)  is  faced  with  a  wide  assortment 
of  risk  types  in  a  program.  Identifying  risk  in  a  program 
is  a  vital  step  in  managing  the  potential,  negative  impacts 
of  risk.  Risk  analysis  is  the  "process  of  examining  each 
identified  risk  area  to  refine  the  description  of  the  risk, 
isolating  the  cause,  and  determining  the  effects." 
(Guidelines  for  Successful  Acquisition  of  Software - 
Intensive  Systems  (GSAM) ,  2000,  p.  6-18)  Before  risk 

analysis  and  mitigation  can  be  discussed,  several  types  of 
risks  that  programs  often  face  must  be  analyzed. 

Sources  of  risk  can  be  generally  classified,  but  are 
not  limited  to,  one  of  the  following  categories:  technical 
risk,  requirements  risk,  schedule  risk  and  cost/funding 
risk.  (Risk  Management  Guide  for  DoD  Acquisition,  2001,  p. 
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a.  Technical  Risk 

Technical  risk  is  the  "degree  to  which  the 
technology  proposed  for  the  program  has  been  demonstrated 
as  capable  of  meeting  all  of  the  program's  objectives." 
(Risk  Management  Guide  for  DoD  Acquisition,  2001,  p.  8) 
Technical  risk  refers  to  the  maturity  level  of  technology 
utilized  in  the  system  being  developed.  The  main  concern 
with  technical  risk  is  that  the  system  will  fail  to  perform 
to  expected  standards  because  of  immature  or  poorly 
integrated  technology.  In  software  development,  a  great 
technical  risk  lies  in  the  difficulty  in  measuring 
developmental  progress  through  the  use  of  Technical 
Performance  Measurements  (TPM) .  TPMs  are  metrics  that  a  PM 
may  use  to  measure  progress  in  a  program.  Many  TPMs  used 
in  DoD  lend  themselves  to  physical  measurements:  weight, 
height,  voltage,  power,  etc.  Given  modern  systems' 
reliance  on  software  to  achieve  technical  objectives,  an 
inability  to  accurately  monitor  software  development 
progress  by  means  of  a  concrete  TPM  will  continue  to  pose  a 
great  technical  risk  to  a  developmental  program. 

jb.  Requirements  Risk 

The  requirements  generation  process  produces 
information  for  decision  makers  on  the  projected  mission 
needs  of  the  war  fighter.  A  system  evolves  from  the 
President's  National  Security  Strategy  (NSS) ,  DoD' s 
National  Military  Strategy  (NMS) ,  through  several  layers  of 
analysis  and  refinement  until  the  issuance  of  the  Mission 
Needs  Statement  (MNS) .  The  MNS  defines,  in  broad,  general 
terms  a  deficient  operational  capability  based  on  threat 


assessments . 


Mission  needs  are  defined  in  broad  operational 
terms  in  a  Mission  Needs  Statement  (MNS)  document.  Based 
on  the  MNS,  services  conduct  Analyses  of  Alternatives  (AoA) 
to  assess  the  potential  for  application  of  fielded,  DoD 
systems  to  meet  the  emergent  requirement.  If  no  suitable 
alternative  exists  within  DoD,  an  Operational  Requirements 
Document  (ORD)  is  issued  which  initiates  the  development  of 
a  new  system.  (Systems  Engineering  Fundamentals,  2001,  p. 
45)  Requirements  definition  is  vital  to  establishing  and 
adhering  to  a  strict  timeline  or  schedule  for  the  program. 

The  Requirements  Generation  Process  is  one  of 
three  elements  in  the  DoD's  principal  decision  support 
system.  The  system  results  in  "identifying  and  documenting 
war  fighting  needs  based  on  current  or  future  mission 
deficiencies  or  technological  opportunities."  (Systems 
Engineering  Fundamentals,  2001,  p.  27)  Figure  1 
illustrates  the  evolving  Requirements  Generation  Process 
from  the  issuance  of  the  ORD  through  system  fielding. 


Figure  1.  Requirements  Generation  Process,  from 

(Test  and  Evaluation  Management  Guide,  2001)  . 
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Ensuring  that  a  system' s  requirements  can  be 
identified  and  established  early  and  accurately  greatly 
reduces  the  risk  of  requirements  creep.  Figure  2 
illustrates  how  the  Requirements  Generation  Process 
overlaps  with  Acquisition  Management  and  the  Planning, 
Programming  and  Budgeting  System  (PPBS)  and  is  a  crucial 
element  to  the  system  development  process. 


Figure  2.  Three  DOD  Decision  Support  Systems, 

from  (CJCSI,  2001). 

The  risk  associated  with  a  requirement  is  linked 
to  the  variability  of  the  requirement.  "Creeping"  or 
changing  requirements  can  lead  to  schedule  delays  and  can 
significantly  impact  a  program.  Requirements  risk  is  the 
"sensitivity  of  the  program  to  uncertainty  in  the  system 
description  and  requirements."  (Risk  Management  Guide  for 
DoD  Acquisition,  2001,  p.  7) 

The  ORD  is  reviewed  several  times  throughout  the 
life  of  a  program.  Each  review  may  alter  original 
requirements,  which  can  initiate  time  consuming  and  costly 
Engineering  Change  Proposals  (ECP ) .  Such  changes  can 
negatively  impact  a  program's  cost  and  schedule.  Figure  3 
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shows  the  interface  between  the  system  lifecycle  and  the 
requirements  analysis  process. 
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Figure  3.  Current  Requirements  and  Acquisition 

Interface,  from  (CJCSI,  2001)  . 


The  current  requirements  and  acquisition 
interface  contains  significantly  fewer  opportunities  to 
impact  a  program  based  on  creeping  mission  requirements 
than  the  previous  acquisition  process;  however,  changing 
requirements  at  any  time  introduces  the  risk  of  costly 
design  changes.  Design  changes  late  in  a  program's  life 
can  be  technologically  challenging  and  costly  to  the 
Government . 

Requirements  risk  may  occur  as  a  result  of  any  of 
the  following: 

•  Operational  requirements  not  properly  established 
or  vaguely  stated  for  program  phase 

•  Requirements  are  not  stable 

•  Required  operating  environment  not  described 

•  Requirements  do  not  address  logistics  and 
suitability 

•  Requirements  are  too  constrictive-identify 
specific  solutions  that  force  high  cost  (GSAM, 
2000,  p.  6-29) 
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Without  adequate  and  stable  requirements 
definition  early  in  the  life  of  the  program,  the  Program 
Management  Office  (PMO)  may  be  forced  to  make  costly 
changes  in  the  system. 

c.  Schedule  Risk 

Program  Managers  are  evaluated  on  the  cost, 
schedule,  and  performance  of  their  program.  (DoD  5000.2  -R, 
2002,  pp.  21,  24)  Many  acquisition  programs  are  driven  by 
time,  or  schedule,  rather  than  by  significant  events  or 
milestones  in  the  program's  progress.  Many  factors  can 
influence  a  program' s  ability  to  adhere  to  a  specific 
schedule.  Schedule  risk  is  the  "adequacy  of  the  time 
allocated  for  performing  the  defined  tasks,  e.g., 
developmental,  production,  etc.  This  factor  includes  the 
effects  of  programmatic  schedule  decisions,  the  inherent 
errors  in  the  schedule  estimating  technique  used,  and 
external  physical  constraints."  (Risk  Management  Guide  for 
DoD  Acquisition,  2001,  p.  8) 

Virtually  every  risk  area  can  degrade  a  program' s 
ability  to  maintain  a  schedule.  In  the  design  of  a  system, 
reliance  on  immature  technology  or  an  unproven  development 
process  can  cause  a  program's  schedule  to  slip.  If 
logisticians  are  not  involved  in  the  early  system 
development  process,  inadequate  supportabil ity  late  in 
development  or  after  fielding  can  result  in  the  necessity 
to  make  engineering  changes  causing  delays  in  the  system' s 
Initial  Operational  Capability  (IOC) . 

Pressure  exists  for  a  PM  to  establish  an 
acquisition  lifecycle  schedule  early  in  the  system's  life 
and  to  maintain  that  schedule  throughout.  Development  time 
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estimates  are  based  on  several  factors  including  parallel, 
or  like-system  development  and  contractor  estimates. 
Department  of  Defense  policy  concerning  acquisition 
schedule  is  as  follows: 

Schedule  parameters  shall  minimally  include  (in 
Acquisition  Program  Baseline  (APB) )  dates  for 
program  initiation,  major  decision  points,  and 
the  attainment  of  initial  operating  capability 
(IOC)  .  The  PM  may  propose,  for  Milestone 
Decision  Authority  (MDA)  approval,  other, 
specific,  critical,  system  events,  as  necessary. 

(DoD  5000. 2-R,  2002,  p.  22) 

A  program' s  risk  of  experiencing  a  schedule  delay 
is  compounded,  for  example,  by  the  development  and 
integration  of  new  technologies,  changing  requirements  and 
budget  constraints,  among  others.  A  program  unable  to 
comply  with  an  approved  schedule  may  risk  cancellation. 

d.  Cost /Funding  Risk 

Without  funding,  a  program  has  no  life.  A 
detailed,  total  ownership  cost  (TOC)  estimate  is  required 
upon  initiation  of  a  program.  DoD  guidelines  are  specific 
in  their  direction  concerning  the  establishment  of  detailed 
cost  estimates: 

Cost  parameters  shall  identify  TOC  (broken-out 
into  direct  costs:  research,  development,  test, 
and  evaluation  costs,  procurement  costs,  military 
construction  costs,  operating  and  support  costs 
(to  include  environmental,  safety,  and 
occupational  health  compliance  costs) ,  and  the 
costs  of  acquisition  items  procured  with 
operations  and  maintenance  funds,  if  applicable. 

Cost  figures  shall  reflect  realistic  estimates  of 
the  total  program,  including  a  thorough 
assessment  of  risk.  (DoD  5000. 2-R,  2002,  p.  23) 

A  PM  clearly  needs  to  provide  an  accurate,  sound 
estimate  on  the  TOC  of  the  program  before  system 
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development  begins.  The  risk  is  the  inability  to 
accurately  predict  costs  given  uncertainty  at  the  outset  of 
a  program.  The  cost  risk  area  is  whether  a  program  has 
"the  ability  to  achieve  the  program' s  life  cycle  cost 
objectives.  This  includes  the  effects  of  budget  and 
affordability  decisions  and  the  effects  of  inherent  errors 
in  the  cost  estimating  technique  (s)  used  (given  that  the 
technical  requirements  were  properly  defined)."  (Defense 
Acquisition  University,  2001,  p.  8) 

Technical,  requirement,  schedule  and  cost/funding 
risks  are  but  a  few  examples  of  many  risk  areas  prevalent 
in  defense  acquisitions.  To  summarize  the  impact  of  each 
risk  area  and  the  interrelatedness  of  each,  the  Government 
Accounting  Office  (GAO)  wrote  in  a  report  regarding 
acquisition  risk: 

Once  in  a  product  development  environment, 
external  pressures  to  keep  the  program  moving 
(such  as  preserving  cost  and  schedule  estimates 
to  secure  budget  approval)  become  dominant.  If  a 
program  manager  decided  that  an  additional  year 
was  needed  to  reach  the  desired  level  of 
technical  maturity  during  the  risk 
reduction/concept  demonstration  phase,  the 
planned  start  of  the  engineering  and 
manufacturing  development  phase  could  be  delayed. 

This  delay  could  jeopardize  funding  for  that 
phase,  thus  risking  the  funding  support  for  the 
entire  program.  (United  States  General  Accounting 
Office,  2000,  p.  16) 

Programs  exist  because  a  need  or  requirement 
exists  to  better  support  or  equip  war  fighters.  All  the 
numerous  risks  surrounding  a  defense  acquisition  program 
threaten  the  DoD' s  ability  to  respond  to  a  specific  mission 
need  or  leverage  emerging  technologies  and  improve  our 
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it  is 


current  war  fighting  capabilities.  Therefore, 
imperative  that  program  managers  actively  manage  and 
mitigate  risks  in  programs.  The  next  section  discusses 
risk  management  techniques  in  DoD. 

C.  RISK  MANAGEMENT 

While  risk  is  the  probability  of  a  future  event 
occurring  and  the  impact  of  that  event,  risk  management  is 
concerned  with  "the  outcome  of  future  events  and  how  to 
deal  with  uncertainty."  (Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  1)  Throughout  DoD,  risk  management 
is  recognized  as  a  vital  management  tool  that  spans  the 
entire  acquisition  lifecycle  from  concept  exploration  to 
operations  and  support.  (GSAM,  2000,  p.  6-4)  If 

implemented  early  into  a  program's  management,  risk 
management  becomes  a  way  of  life.  The  key  to  successfully 
managing  risk  is  planning  and  forward  thinking. 

To  support  these  efforts,  assessments  should  be 
performed  as  early  as  possible  in  the  life  cycle 
to  ensure  that  critical  technical,  schedule  and 
cost  risks  are  addressed  with  mitigation  actions 
incorporated  into  program  planning  and  budget 
projections.  (Defense  Systems  Management  College, 

2001,  p.  2-3) 


The  remainder  of  this  section  will  discuss  risk 
management  practices  and  techniques  commonly  used  in  DoD. 


This 

thesis  will  break 

down 

and 

analyze 

risk 

management 

in  four  parts:  (1) 

Risk 

Planning, 

(2) 

Risk 

Assessment, 

(3)  Risk  Mitigation, 

and 

(4) 

Risk 

Tracking . 

(Defense  Acquisition  Deskbook  (DAD),  2002) 
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A  Continuous  Interlocked  Process — Not  an  Event 


Figure  4.  Risk  Management  Continuum,  from  (DAU, 

Systems  Engineering  Fundamentals,  2001) . 

1 .  Risk  Planning 

Risk  planning  is  the  process  of  "developing  and 
documenting  an  organized,  comprehensive  and  interactive 
strategy  and  methods  for  identifying  and  tracking  risk 
areas,  developing  risk-mitigation  plans,  performing 
continuous  risk  assessments  to  determine  how  risks  have 
changed  or  what  new  risk  exists."  (GSAM,  2000,  p.  6-11) 
Planning  for  adequate  resources  is  vital  to  implementing  a 
risk  management  plan  throughout  the  entire  lifecycle  of  the 
program.  The  DoD  5000. 2-R  mandates  that  PMs  include  risk 
management  in  the  acquisition  strategy. 

Risk  planning  is  a  continuous  effort  throughout  the 
life  of  a  program.  Risk  planning  is  not  a  single  event. 
(Systems  Engineering  Fundamentals,  2001,  p.  134)  Initial 
planning  includes  "establishing  a  strategy;  establishing 
goals  and  objectives;  planning  assessment,  handling  and 
monitoring  activities;  identifying  resources,  tasks  and 
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responsibilities  and  establishing  a  method  to  document  and 
disseminate  information  on  a  continuous  basis."  (Systems 
Engineering  Fundamentals,  2001,  p.  135)  An  example  Risk 
Management  Plan  (RMP)  outline  is  depicted  in  Figure  5 
below : 


Introduction 
Program  Summary 
Definitions 

Risk  Management  Strategy  and  Approach 
Organization 

Risk  Management  Process  and  Procedures 
Risk  Planning 
Risk  Assessment 
Risk  Handling 
Risk  Monitoring 

Risk  Management  Information  System,  Documentation  and  Reports 


Figure  5.  Risk  Management  Plan  Outline/Format, 
from  (Systems  Engineering  Fundamentals,  2001) . 

An  important  aspect  of  risk  planning  is  the 
identification  of  shortfalls,  whether  technical  expertise 
or  resources.  Identifying  shortfalls  allows  a  program 
office  to  identify  risk  areas  that  may  require  additional 
augmentation  or  tracking.  The  RMP  should  be  fully 
integrated  into  the  program  Acquisition  Strategy.  Within 
the  framework  of  the  Integrated  Product  and  Process 
Development  (IPPD)  concept,  assigning  a  Risk  Management 
Coordinator  (RMC)  to  a  program  office  provides  the  team  a 
focal  point  for  risk  management  who  is  responsible  for 
implementing  and  supervising  the  risk  management  process. 

Once  the  RMP  has  addressed  the  risk  management 
strategy  and  organization,  the  next  step  is  to  identify  or 
assess  program  risks.  The  next  section  will  discuss  Risk 
Identification/Asses  sment . 
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2.  Risk  Identification/Assessment 

Risks  can  be  viewed  as  opportunities.  "Risk  and 
opportunity  go  hand  in  hand.  Success  cannot  be  achieved 
without  some  degree  of  risk."  (Carnegie  Mellon  Software 
Engineering  Institute  (SEI),  1999,  p.  3)  Opportunities  are 
defined  as  "chances  for  progress  or  advancement"  or 
"chances  for  improving  the  value  of  the  project  results." 
(Forsberg,  Mooz,  Cotterman,  2000,  p.  188)  Programs  and 
PM' s  risk  failure  when  they  fail  to  identify  program  risks. 

One  of  the  biggest  problems  a  project  manager 
faces  is  motivating  team  members  to  identify 
risks.  You  want  to  make  everyone  risk  conscious. 
However,  there  is  often  that  hesitancy  to  surface 
risks,  lest  one  be  labeled  a  worrier  or  negative 
thinker.  You  can't  mitigate  it  (risk)  if  you 
don't  know  it's  there  so  it's  better  to 
anticipate  a  lot  of  problems,  some  of  which  won't 
happen,  than  too  few  and  miss  the  "project 
killers."  (Forsberg,  Mooz,  Cotterman,  2000,  p. 

193) 

This  section  discusses  risk  identification  and 
assessment . 

Risk  identification  begins  by  compiling  the  program' s 
risk  events.  Examining  each  Work  Breakdown  Structure  (WBS) 
product  and  process  element  in  terms  of  the  sources  or 
areas  of  risk  most  easily  identifies  risk  events.  (Systems 
Engineering  Fundamentals,  2001,  p.  11) 

A  WBS  is  a  "means  of  organizing  system  development 
activities  by  examining  the  physical  and  architectural 
qualities  of  a  system."  (Systems  Engineering  Fundamentals, 
2001,  p.  85)  The  WBS  enables  PM' s  to  identify  potential 
risk  areas  in  development  and  system  integration. 
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Figure  6.  Example  Work  Breakdown  Structure,  from 

(Systems  Engineering  Fundamentals,  2001)  . 


The  WBS  enables  an  entire  system  to  be  visualized 
through  a  "logical  breakdown  of  product  elements  into  work 
packages."  (Systems  Engineering  Fundamentals,  2001,  p.  86) 
Once  risk  areas  are  identified,  a  PM  needs  to  categorize 
and  prioritize  risks  elements  in  the  program, 
a.  Risk  Analysis  Process 

Through  the  IPPD  process,  program  planners, 
engineers,  logisticians  and  other  functional  area 
representatives  discuss  and  analyze  identified  risk  areas. 
The  analysis  includes  determining  the  likelihood  that  a 
risk  area  will  occur  and  the  impact  of  the  occurrence. 
Many  tools  exist  to  assist  in  this  process.  A  risk  matrix 
is  a  helpful  tool  to  assess  risk  areas  in  programs. 
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ASSESSMENT  GUIDE 
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The  Risk  WII  Happen? 
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b 
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c 
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d 
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1  2  3  4  5 

Consequence 
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LOW  -  Mnrnim  Import 
Normal  overact  needed  to 
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>10% 


None 
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Moderate  Impact 

Major  Impact 
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Figure  7.  Example  Risk  Matrix,  from  (GSAM,  2000)  . 


The  matrix  enables  a  PMO  to  individually  analyze 
risk  areas,  determine  the  likelihood  of  occurrence  and 
assess  the  impact  of  the  risk  on  the  program's  cost, 
schedule  or  performance. 

Risk  assessments  are  categorized  green,  amber  or 
red  in  ascending  criticality.  A  PMO  may  elect  to  pay 
greater  attention  to  amber  or  red  items  throughout  the  risk 
mitigation  process  than  low  risk,  or  "green"  risks. 
Critical,  or  red,  risks  may  be  deemed  unacceptable  to  a 
program  and  generate  engineering  change  proposals  (ECP)  or 
an  aggressive  mitigation  strategy  to  reduce  the  risk  to  an 
acceptable  level. 

The  assessments  for  probability  of  risk 
occurrence  are  based  on  program  office  personnel 
experience,  similar  or  parallel  system  development. 
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modeling  and  simulation,  like-component  mean  time  to 

failure  and  technology  maturity,  among  others.  The  risk 
impact  assessment  is  similarly  determined.  The  following 
provides  examples  of  risk  impact  determination  criteria: 

•  Comparisons  with  similar  systems, 

•  Relevant  lessons-learned  studies, 

•  Experience, 

•  Results  from  tests  and  prototype  development, 

•  Data  from  engineering  or  other  models, 

•  Specialist  and  expert  judgments, 

•  Analysis  of  plans  and  related  documents, 

•  Modeling  and  simulation, 

•  Sensitivity  of  analysis  of  alternatives.  (Risk 

Management  Guide  for  DoD  Acquisition,  2001,  p. 

15) 

Once  the  risk  area  probabilities  and  impacts  are 
determined,  the  risks  may  be  prioritized  and  rated  based  on 
greatest  probability  of  occurrence  and  impact  to  the 

program. 

b.  Risk  Rating  and  Prioritization 

Risk  ratings  are  indications  of  potential  impact 
of  risks  on  a  program.  Risks  are  often  rated  and 
categorized  as  High,  Moderate  or  Low.  Risk  ratings  and 
prioritization  are  considered  an  integral  part  of  risk 
analysis.  Prioritizing  risks  is  the  first  step  in 

developing  a  risk  mitigation  strategy,  focusing  efforts 
first  on  risks  that  carry  the  greatest  potential  impact  on 
the  program.  Several  tools  exist  to  assist  the  PMO  to  make 
preliminary  judgments  regarding  risk  classification. 
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Figure  8.  Example  Risk  Classification  Matrix, 
from  (Systems  Engineering  Fundamentals,  2001)  . 

The  documented,  prioritization  is  called  a  risk 
Watch  List.  (Risk  Management  Guide  for  DoD  Acquisition, 
2001,  p.  17)  A  prioritized  watch  lists  allows  the  PMO  to 
visualize  risk  areas  and  concentrate  management  and 
leadership  efforts  where  they  are  most  needed. 


Priority 

Area/Source 

Process 

Location 

Risk  Event 

Proba¬ 

bility 

Conse¬ 

quence 

Risk 

Rating 

1 

Design 

WBS  3.1 

Design  not 
completed  on  time 

Very 

likely 

Severe 

High 

2 

Figure  9.  Example  Risk  Rating  Matrix,  from  (Risk 

Management  Guide  for  DoD  Acquisition,  2001)  . 
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Based  on  identified  risk  areas  and  their 
probability  of  occurrence  and  impact  to  the  program,  the 
PMO  can  develop  a  mitigation  strategy. 


Figure  10.  Initial  Risk  Identification  and 

Prioritization,  from  (Risk  Management  Guide  for 
DoD  Acquisition,  2001) . 


3 .  Risk  Mitigation 

Risk  mitigation  is  the  process  that  "identifies, 
evaluates,  selects,  and  implements  options  in  order  to  set 
risk  at  acceptable  levels  given  program  constraints  and 
objectives."  (GSAM,  2000,  p.  6-19)  Risk  mitigation 
includes  determining  what  should  be  done  to  manage  a 
particular  risk,  how  often  it  should  be  done  and  reported, 
who  is  responsible  for  handling  it  and  what  the  cost  impact 
of  managing  the  risk  is.  PM' s  must  determine  the  possible 
"consequences  of  action  or  inaction  as  well  as  conducting  a 
cost-benefit  analysis  of  mitigation  actions."  (GSAM,  2000, 
p.  6-20)  Risk  mitigation  actions  should  also  be  closely 
tied  to  metrics  that  measure  the  success,  progress  or 
failure  of  a  particular  mitigation  action. 
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A  program' s  RMC  has  several  options  regarding  risk 
handling.  RMCs  may  assess  risk  mitigation  proposals  based 
on  the  following  criteria: 

•  Can  the  option  be  feasibly  implemented  and  still 
meet  the  user's  needs? 

•  What  is  the  expected  effectiveness  of  the 
handling  option  in  reducing  program  risk  to  an 
acceptable  level? 

•  Is  the  option  affordable;  based  on  both  fiscal 
and  time  constraints? 

•  What  effect,  if  any,  does  the  option  have  on  the 
system' s  technical  performance?  (Risk  Management 
Guide  for  DoD  Acquisition,  2001,  p.  19) 

Based  on  the  assessments,  the  PMO  may  choose  among 
several  risk  mitigation  (i.e.  handling)  techniques. 

a.  Risk  Avoidance 

A  PMO  may  avoid  a  risk  of  one  alternative  by 
choosing  another,  less  risky  alternative.  This  process  is, 
in  essence,  a  method  to  reduce  risk  since  it  does  not 
completely  eliminate  risk.  An  important  distinction  to 
make  is  that  risk  avoidance  must  be  a  conscious  decision  to 
choose  lower  versus  higher  risk  options.  Avoiding  risk  by 
ignoring  its  presence  and  potential  impact  is  an 
unacceptable  solution. 

Risk  avoidance  may  be  done  in  parallel  with  "the 
up-front  requirements  analysis,  supported  by  a  cost  per 
requirement  trade  study.  The  concept  of  Cost  as  an 
Independent  Variable  (CAIV)  is  an  example  of  such  a  study. 
It  is  imperative  that  user  representatives  are  present 
during  any  trade-off  study  or  decision."  (Risk  Management 
Guide  for  DoD  Acquisition,  2001,  p.  20) 
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Risk  cannot  be  altogether  avoided.  Remembering 
that  a  risk  represents  an  opportunity,  risk  aversion  can 
lead  to  a  poor  managerial  environment . 

A  risk-averse  culture  inhibits  risk  management 
more  than  does  the  lack  of  a  management 
infrastructure  or  a  repeatable  method.  Such  a 
culture  generally  rewards  crisis  management  and 
punishes  those  who  identify  why  the  project  may 
not  succeed.  (GSAM,  2000,  p.  6-19) 

It  is  evident  that  the  avoidance  of  one  risk  in 
favor  of  another,  less-risky  alternative  is  not  the  same  as 
attempting  to  eliminate  risk  from  a  program  altogether. 

b.  Risk  Control 

Risk  may  be  controlled  through  the  continuous 
monitoring  and  correction  of  risky  conditions.  Risk 

control  "monitors  and  manages  the  risk  in  a  way  that 
reduces  the  probability  and  impact  of  its  occurrence  on  the 
program."  (Risk  Management  Guide  for  DoD  Acquisition,  2001, 
p.  19)  Risk  control  involves  reviews,  inspections,  risk 
milestone  reviews,  development  of  fallback  positions  and 
similar  management  techniques.  Controlling  risk  involves 
"developing  a  risk  reduction  plan  and  then  tracking  to  that 
plan."  (GSAM,  2000,  p.  6-20)  The  following  lists  examples 
of  risk  control  actions: 

•  Multiple  development  efforts 

•  Alternative  designs 

•  Early  prototyping 

•  Incremental  development 

•  Technology  maturation  efforts. 

•  Use  of  mock-ups,  and 

•  Modeling  and  simulation  (Risk  Management  Guide 

for  DoD  Acquisition,  2001,  p.  19-20) 
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While  this  is  not  an  exhaustive  list,  it  provides 
examples  of  risk  control  actions  while,  again,  not 
eliminating  risk.  All  of  these  methods  reduce  unnecessary 
risks  while  working  to  meet  user  requirements. 

c.  Risk  Assumption 

Risk  assumption  involves  a  conscious  decision  to 
accept  a  risk  level  and  potential  impact  of  occurrence 
without  taking  any  steps  to  manage  or  reduce  the  risk.  The 
challenge  for  PMs  lies  in  determining  an  acceptable  level 
of  risk.  Risk  assumption  is  best  reserved  for  low-level 
risks,  in  terms  of  impact,  or  risks  whose  probability  of 
occurrence  is  remote.  Whenever  possible,  PMOs  will  handle 
risk  assumptions  by  ensuring  that  a  contingency  plan  is  in 
place  to  address  and  handle  emerging  risks  previously 
assumed  in  the  program.  A  management  reserve,  additional 
funds,  personnel  or  schedule  time,  must  be  in  place  to 
accomplish  contingency  management  actions.  (Risk  Management 
Guide  for  DoD  Acquisition,  2001,  p.  21) 

d.  Risk  Transference 

Risk  transference  involves  more  than  one  entity 
sharing  risk,  which  is  often  cost  risk.  This  technique  is 
frequently  used  between  the  Government  and  contractors. 
The  Government  provides  a  contractor  financial  incentives 
(award  fees,  contractual  incentives),  for  example,  to  share 
in  managing  risk.  A  contract  between  the  Government  and  a 
prime  contractor  generally  initiates  the  risk  transference 
process.  The  Government  may  provide  financial  incentives 
to  a  prime  contractor  to  minimize  or  reduce  risks  in 
numerous  risk  areas  to  include  system  technical 
performance,  development  cost  and  adherence  to  schedule. 


27 


This  thesis  discusses  risk  transference  through  the 
contracting  process  in  subsequent  chapters. 

4 .  Risk  Tracking /Monitoring 

Risk  monitoring  is  the  continuous  process  of  "tracking 
and  evaluating  the  risk  management  process  by  metric 
reporting,  enterprise  feedback  on  watch  list  items  and 
regular  enterprise  input  on  potential  developing  risk 

areas."  (Systems  Engineering  Fundamentals,  2001,  p.  139) 
The  process  involves  evaluating  how  current  and  past  risk 
handling  actions  compare  with  previously  established  risk 
management  metrics.  Program  metrics  are  used  for  formal 
analyses  of  how  well  the  various  development  processes  are 
progressing  in  comparison  to  TPMs,  schedule  predictions, 
technology  maturity,  etc. 

The  purpose  of  monitoring  and  tracking  risk  is  two¬ 

fold.  First,  to  determine  if  risk  elements  are  in  danger 
of  adversely  affecting  cost,  schedule  or  performance  of  the 
program.  Second,  risk  monitoring  aids  in  identifying  risk 
areas  not  initially  identified  and  assessed.  The  "Goal, 
Question,  Metric  paradigm"  (GQM)  is  a  simple  example  of  the 
risk  tracking/monitoring  process.  (GSAM,  2001,  p.  6-21) 

The  GQM  method  consists  of  the  following  steps.  The 

first  step  is  to  select  the  goals  of  the  risk  area- 

monitoring  program.  The  second  step  is  to  identify  "the 
questions  that  should  be  asked  to  determine  if  the  goals 
are  being  met."  (GSAM,  2001,  p.  6-21)  The  final  step  is  to 
identify  metrics  or  indicators  that  allow  one  to  answer  the 
question,  "Are  the  goals  being  met?"  (GSAM,  2001,  p.  6-21) 

The  final  step  in  the  risk  tracking/monitoring  process 
is  to  document  the  findings.  A  program  office-wide  shared 
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database  allows  all  PMO  personnel  to  update  risk  area 
progress  and  emergent  threats  in  the  program. 
Documentation  "provides  the  basis  for  program  assessments 
and  updates  as  the  program  progresses."  (Risk  Management 
Guide  for  DoD  Acquisition,  2001,  p.  21)  Proper  risk 
documentation  also  helps  to  incorporate  new  personnel  into 
the  program  office  and  reduces  the  hazard  of  repeating  past 
mistakes.  Depending  on  the  technical  depth  and  size  of  a 
program,  PMs  will  establish  a  standard  list  of  risk 
documentation  to  be  presented  at  established  intervals. 

The  following  list  illustrates  example  reports: 

•  Program  metrics 

•  Technical  reports 

•  Earned  Value  (EV)  reports 

•  Watch  list 

•  Schedule  performance  report 

•  Critical  risk  process  reports  (Risk  Management 

Guide  for  DoD  Acquisition,  2001,  p.  22) 

The  above  list  provides  examples  of  reports  that  may  be  used  to 

document  the  implementation  of  the  RMP  to  assess  its  successes 

or  shortfalls.  The  next  section  will  analyze  several  techniques 

that  DoD  can  use  to  manage  risk  in  developmental  systems. 

D.  DEPARTMENT  OF  DEFENSE  RISK  MANAGEMENT  TECHNIQUES 

Many  risk  management  techniques  are  available  to  the 
DoD.  The  DoD  5000. 2 -R  requires  PMs  to  ensure  that 
contractors'  management  information  systems  used  in 
"planning  and  controlling  contract  performance  meet  the 
Earned  Value  Management  Systems  (EVMS)  guidelines."  (DoD 
5000. 2-R,  p.  49)  The  Program  Milestone  Decision  Authority 
(MDA)  may  waive  the  requirement,  in  some  instances.  Other 
than  EVMS,  no  particular  risk  management  technique  is 
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mandatory  in  DoD .  This  section  discusses  several  risk 
management  options  available  to  PMs . 

1.  Test  and  Evaluation  (T&E) 

Periodic  Test  and  Evaluation  (T&E)  events  early  and 
throughout  a  program's  development  are  a  method  of 
evaluating  the  progress  and  technological  maturity  of  a 
system  and  identifying  new  risk  areas.  A  test  plan  is  a 
risk  reduction  method  if  implemented  early.  The  T&E 
process  is  "an  integral  part  of  the  systems  engineering 
process  which  identifies  levels  of  performance  and  assists 
the  developer  in  correcting  deficiencies."  (Test  and 
Evaluation  Management  Guide,  2001,  p.  1-1) 

T&E  is  an  important  risk  management  technique  because 
it  helps  developers  and  managers  evaluate  levels  of 
technical  performance,  reliability,  maintainability, 
technical  maturity  and  cost  and  schedule  conformance.  T&E 
is  a  proactive  measure  that  validates  earned  levels  of 
performance  and  identifies  emerging  risks  so  they  may  be 
managed  and  tracked: 

Correcting  defects  in  weapons  has  been  estimated 
to  add  from  10-30%  to  the  cost  of  each  item. 

Such  costly  redesign  and  modification  efforts  can 
be  reduced  if  carefully  planned  and  executed  test 
and  evaluation  programs  are  used  to  detect  and 
fix  system  deficiencies."  (Test  and  Evaluation 
Management  Guide,  2001,  p.  1-1) 

T&E,  though  often  costly  and  time-consuming  to 
perform,  has  the  potential  to  help  control  costs  and  ensure 
a  desired  level  of  system  performance  in  the  long  run  of 
the  program.  Figure  11  illustrates  the  relationship 
between  committing  program  dollars  to  thoroughly  test  and 
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evaluate  the  system  incrementally  throughout  its  life  and 
the  life-cycle  cost  of  the  system. 


Figure  11.  Life-Cycle-Cost  Decision  Impact  and 

Expenditures,  from  (Test  and  Evaluation 
Management  Guide,  2001) . 

The  figure  demonstrates  that  a  system  that  is  not 
properly  tested  early  during  its  life  cycle  may  incur  far 
greater  Operations  and  Support  (O&S)  costs  than  a  system 
that  undergoes  a  thorough  T&E  plan  to  identify  and  manage 
risks  early  throughout  system  development  and  demonstration 
(SDD)  . 

T&E  also  serves  as  a  decision-making  tool  for  senior 
leaders  in  DoD.  T&E  events  are  required  before  a  system 
can  undergo  a  Milestone  Review.  Figure  12  illustrates  the 
relationship  between  T&E  and  the  Acquisition  Process. 

The  Test  and  Evaluation  Master  Plan  (TEMP)  is  written 
as  a  part  of  the  formal  Acquisition  Strategy  pending  a 
Milestone  B  decision  authorizing  entry  into  System 
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Development  &  Demonstration  (SDD)  .  The  TEMP  addresses 
system  items  to  be  tested  as  well  as  laying  out  the 
Integrated  Test  Program  (ITP)  Schedule.  The  TEMP  is 
updated  continuously  throughout  the  program  and  officially 
at  each  Acquisition  Milestone. 


Figure  12.  Testing  and  the  Acquisition  Process, 

from  (Test  and  Evaluation  Management  Guide, 

2001)  . 

The  TEMP  updates  include  guidance  from  the  MDA  on 
testing  areas  of  interest  during  the  follow-on  acquisition 
phase.  Testing  areas  focus  on  validating  system 

capabilities  and  detecting  and  reporting  of  "deficiencies 
that  may  adversely  impact  the  performance  capability  or 
availability/supportability  of  a  system."  (Test  and 
Evaluation  Management  Guide,  2001,  p.  1-4) 

In  summary,  T&E  is  "the  discipline  that  helps  to 
illuminate  risk  areas  of  vulnerability."  (Test  and 
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Evaluation  Management  Guide,  2001,  p.  1-7)  A  rigorous  T&E 
program  can  identify  and  manage  program  risks  in  a  manner 
that  saves  time  and  money,  while  also  ensuring  that  the 
tester  provides  the  user  timely  and  cost  effective  answers 
to  operational  requirements. 

2.  Cost  as  an  Independent  Variable  (CAIV) 

Cost  as  an  Independent  Variable  (CAIV)  is  the  process 
of  balancing  cost,  schedule,  performance  and  risk  early  in 
a  systems  development  in  order  to  manage  a  program  to  a 
cost  objective."  (Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  29)  CAIV  involves  a  joint  PMO-user 
representative  trade-off  analysis  between  system 
performance  and  program  costs.  The  underlying  premise  of 
CAIV  is  that  "if  costs  are  too  great,  and  there  are  ways  to 
reduce  them,  then  the  user  and  developer  may  reduce 
performance  requirements  to  meet  cost  objectives."  (Risk 
Management  Guide  for  DoD  Acquisition,  2001,  p.  30)  Risk 
assessments  are  essential  in  the  CAIV  trade-off  process. 

Assessing  risk  areas  and  identifying  cost  drivers 
provide  PMs  and  user  representatives  with  data  that  can  be 
used  when  conducting  trade-offs  between  system  performance 
and  cost.  The  concept  of  CAIV  is  that  "equal  emphasis  must 
be  placed  on  managing  cost  and  schedule  risks"  as  it  is  on 
system  technical  risk.  (Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  30) 

3 .  Earned  Value  Management  (EVM) 

The  Earned  Value  Management  System  (EVMS)  is  a  joint 
DoD-Industry  agreement  established  in  1995  that  details  DoD 
5000. 2-R  contractor  requirements  with  respect  to  the 
implementation  of  Earned  Value  Management  (EVM) .  EVM  is  a 
process  that,  through  one  hundred  percent  system 
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evaluates 


decomposition  and  definition,  evaluates  a  program' s 
progress  in  terms  of  cost  and  schedule.  EVM  can  be  used  as 
an  "early  warning  signal"  to  a  PM  to  identify  risks  of 
overrunning  cost  or  schedule  constraints.  (Earned  Value 
Project  Management,  September  2002) 


By  decomposing  a  system' s  requirements  and  defining 
the  system  thoroughly,  managers  can  provide  cost  estimates 
per  development  function  and  track  the  program's  progress. 
PMs  periodically  assess  actual  costs  to  date  versus 
projected  costs  and  actual  time  requirements  versus 
projected  time  to  determine  variances.  The  identification 
of  variance  can  help  identify  new  or  underestimated  risk 
areas  and  alert  the  PM  to  take  action  to  assess  and 
mitigate  the  cost  or  schedule  risks.  Figure  13  illustrates 
that  a  program's  progress  may  be  evaluated  after  only  15% 
completion  of  the  program. 

The  key  to  using  EVM  effectively  is  an  accurate 
program  process  definition  and  decomposition.  When  used 
properly,  EVM  affords  a  PM  visibility  on  a  program's  cost 
and  schedule  status.  The  PM  can  then  make  necessary 
changes  or  perform  trade-off  studies  to  meet  cost  and 
schedule  thresholds.  EVM  is  an  effective  technique  to 
combat  cost  and  schedule  risks. 
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Figure  13.  Earned  Value  Management  As  a  Risk 

Management  Technique,  from  (Earned  Value  Project 
Management,  September  2002) . 

4 .  Modeling  and  Simulation 

Modeling  and  simulation  (M&S)  may  be  used  by  a  PMO  to 
manage  risk  throughout  the  entire  life  cycle  of  a  system. 
Models  and  simulations  can  "reduce  time,  resources,  and 
acquisition  risk"  and  may  contribute  to  increasing  the 
system's  overall  quality  and  performance.  (Risk  Management 
Guide  for  DoD  Acquisition,  2001,  p.  27)  In  managing  risk, 
M&S  can  assist  in  the  following  ways: 

•  Develop  alternate  concepts  during  system  design 

•  Predict  performance  in  support  of  trade-off 

studies 

•  Evaluate  system  design  and  support  preliminary 
design  reviews 

•  Predict  performance  and  supplement  live  tests 

during  system  testing 

•  Examine  the  military  value  of  the  item 

•  Determine  the  impact  of  design  changes 

•  Hone  requirements 
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•  Develop  life-cycle  support  requirements  and 
assessments  (Risk  Management  Guide  for  DoD 
Acquisition,  2001,  p.  27) 

The  risk  management  techniques  this  thesis  has 
addressed  are  not  mutually  exclusive.  For,  example  M&S  may 
be  used  extensively  during  T&E. 

Modeling  and  Simulation  during  T&E  may  be  used  for 
"concept  evaluation,  extrapolation,  isolation  of  design 
effects,  efficiency,  representation  of  complex  environments 
and  overcoming  inherent  limitations  in  actual  testing." 
(Test  and  Evaluation  Management  Guide,  2001,  p.  14-8)  By 
performing  M&S  during  T&E,  the  PM  may  thoroughly  test  a 
system  under  virtual  conditions  and  environments,  which  may 
otherwise  be  cost-prohibitive.  M&S  helps  to  reduce 
technical  risk  by  discovering  design  effects  on  the  overall 
system  before  physically  incorporated  into  the  system.  M&S 
reduces  schedule  and  cost  risk  by  assisting  engineers  to 
make  the  right  decisions  early  based  on  data  gathered 
during  M&S  tests.  Additionally,  models,  which  prove  to  be 
accurate  predictors  of  actual  test  events  may  allow  the  PM 
to  waive  further,  live  tests  based  on  a  high  degree  of 
confidence  in  the  model's  data. 

Modeling  and  Simulation  is  only  as  good  as  the  data 
and  variables  that  are  inputs  to  the  model.  M&S  can  be  a 
great  risk  management  tool  if  adequate  time  is  taken  to 
ensure  the  accuracy  of  input  data.  The  DoD  5000. 2 -R 
encourages  PMs  to  incorporate  M&S  activities  where 
applicable  to  their  respective  programs  because  of  the 
potential  cost  and  time  reductions  as  well  as  enhanced 
system  development  and  performance  validation. 
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5. 


Including  Risk  Management  in  the  Contracting 
Process 


The  final  risk  management  technique  this  thesis 
addresses  is  the  inclusion  of  risk  management  throughout 
the  contracting  process.  Managing  risk  through  contracting 
often  involves  risk  transference  from  either  the  Government 
to  the  contractor  or  vice  versa.  "By  properly  setting  the 
expectations  of  all  players,  explicitly  agreeing  upon  the 
deliverable  items  produced  by  the  event,  and  securing 
sponsorship  from  project  management,  a  high  degree  of 
success  is  assured."  (SEI,  1999,  p.  17)  The  shift  from 
Military  Specifications  and  Standards  to  Performance  based 
requirements  placed  increased  risk  in  the  hands  of  the 
contractor.  However,  "if  a  program  fails  because  risk 
isn't  managed  well  by  the  contractor,  the  PM  is  ultimately 
responsible."  (Defense  Acquisition  Deskbook  (DAD),  Top 
Eleven  Ways  to  Manage  Technical  Risk,  1998,  p.  3-1) 
Numerous  opportunities  exist  throughout  the  contracting 
process  that  enables  the  Government  to  manage  system  risk. 
The  remainder  of  this  section  discusses  risk  management 
through  the  Request  for  Proposal  (RFP)  and  contract  award 
fee  incentives. 

a.  The  Request  for  Proposal  (RFP) 

Even  before  an  RFP  is  released,  a  PM  should 

conduct  a  preliminary  risk  assessment  to  ensure  that  "the 

program  to  be  described  in  the  RFP  is  executable  within 
technical,  schedule  and  budget  constraints."  (DAD,  Top 
Eleven  Ways  to  Manage  Technical  Risk,  1998,  p.  3-1)  The 

RFP  should  require  offerors  to  address  their  RMP  and 
initial  risk  assessment  and  mitigation  plan  for  moderate  to 
high-risk  areas.  (DAD,  Top  Eleven  Ways  to  Manage  Technical 
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Risk,  1998,  p.  3-1)  The  RFP  should  also  stipulate  that 
offerors  must  make  periodic  risk  assessment  reports  to  the 
Government.  Contractors'  reports  serve  as  input  to  the 
PM' s  risk  monitoring  and  tracking  program.  By  requiring  a 
risk-based  approach,  offerors'  proposals  should  "state  how 
they  would  plan  and  schedule  [software]  activities  based 
upon  realistic  assessments  of  technical  challenges  and 
risks"  so  that  the  Government  may  evaluate  management 
capabilities."  (GSAM,  2000,  p.  6-30)  Whether  the 
development  risk  is  hardware,  software  or  a  combination  of 
both,  the  RFP  is  a  vehicle  to  inject  risk  management 
activities  into  the  program.  The  RFP  contains  several 
sections,  which  allow  the  Government  to  directly  address 
risk  areas  in  the  solicitation. 

Section  C,  Description/Specifications/Statement 
of  Work,  includes  any  descriptions  or  specifications 
required  in  the  offeror's  response.  (DAD,  Top  Eleven  Ways 
to  Manage  Technical  Risk,  1998,  p.  3-2)  Example  Section  C 
wording  that  addresses  risk  is  as  follows: 

The  Offeror  shall  describe  its  proposed  risk 
management  program.  The  Offeror  shall  describe 
how  they  intend  to  identify,  assess,  mitigate, 
and  monitor  potential  technical  risks.  Critical 
technical  risks  which  may  adversely  impact  cost, 
schedule,  or  performance  shall  be  identified 
along  with  proposed  risk  mitigation  methods  for 
all  risks  identified  as  moderate  or  high."  (DAD, 

Top  Eleven  Ways  to  Manage  Technical  Risk,  1998, 
p.  3-2) 

Section  C  allows  the  Government  to  evaluate 
offerors'  RMPs  and  compare  competing  contractors'  responses 
with  respect  to  risk  identification,  assessment  and 
handling.  The  contractor's  detailed  Statement  of  Work 
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(SOW)  submitted  in  response  to  the  RFP  can  inform  the 
Government  as  to  the  level  of  contractor  requirement 
understanding.  The  SOW  may  also  warn  PMOs  of  high-risk 
acquisition  plans  on  the  part  of  the  contractor. 

Section  L,  Instructions ,  Conditions ,  and  Notices 
to  Offerors ,  provides  instructions  to  the  offeror  in 
proposal  preparation.  The  Government  may  elect  to  include 
risk  management  requirements  in  Section  L  as  long  as  the 
risk  items  are  consistent  throughout  the  RFP.  Example 
Section  L  language  is  as  follows: 

The  Offeror  shall  discuss  past/present 
performance  in  the  implementation  of  risk 
reduction/mitigation  efforts  similar  to  those 
proposed  for  the  reduction  of  all  risks 
identified  as  moderate  or  high."  (DAD,  Top  Eleven 
Ways  to  Manage  Technical  Risk,  1998,  p.  3-2) 

Section  L  ensures  that  offerors  include 
information  pertaining  to  risk  that  may  enable  the 
Government  to  evaluate  a  contractor' s  technical  past 
performance  or  ability  to  manage  technical  risk. 

Section  M,  Evaluation  Factors  for  Award,  notifies 
offerors  of  "the  evaluation  factors  against  which  all 
proposals  will  be  evaluated."  (DAD,  Top  Eleven  Ways  to 
Manage  Technical  Risk,  1998,  p.  3-2)  Section  M  should 
focus  on  areas  outlined  for  offerors  in  Section  L  so  that 
the  Government's  instructions  for  proposal  preparation  are 
consistent  with  the  evaluation  criteria.  Section  M  should 
list  the  relative  importance  or  hierarchy  of  evaluation 
criteria  such  as  past  performance,  risk  management, 
technical  performance,  cost,  schedule,  management,  etc. 
The  Government  is  not  required  to  quantify  the  importance 
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or  ranking  of  evaluation  criteria,  but  must  inform  offerors 
which  criteria  will  be  considered  most  in  the  source 
selection  process.  An  example  Section  M  risk  management 
language  follows: 

The  Government  will  evaluate  the  Offeror' s 
proposed  risk  management  program  and  plans  for 
identifying,  assessing,  mitigating,  and 
monitoring  risks,  as  well  as  proposed  plans  for 
mitigating  those  risks  identified  as  moderate  or 
high."  (DAD,  Top  Eleven  Ways  to  Manage  Technical 
Risk,  1998,  p.  3-3) 

Through  specific  risk  management  Section  L 
instructions  and  corresponding  Section  M  evaluation 
criteria,  the  Government  can  ensure  that  only  proposals 
that  demonstrate  responsible  and  capable  risk  area  handling 
are  considered  for  contract  award. 

The  DoD  5000. 2-R  states  that  risk  reduction 
through  the  use  of  mature  technology  will  be  a  significant 
factor  in  the  Source  Selection  Process  (SSP)  .  The  purpose 
of  the  SSP  and  competition  in  contracting  is  to  "select  the 
contractor  whose  performance  can  be  expected  to  meet  the 
Government's  requirements  at  an  affordable  price."  (DAD, 
Top  Eleven  Ways  to  Manage  Technical  Risk,  1998,  p.  3-4) 
Several  vehicles,  including  the  detailed  Statement  of  Work 
(SOW),  aid  the  PM  in  evaluating  solicitation  respondents. 

b.  Statement  of  Objectives  (SOO) 

The  Statement  of  Objectives  (SOO)  is  a  concept 
that  transfers  the  responsibility  for  preparing  the 
Statement  of  Work  (SOW)  from  the  Government  to  the  offeror. 
(DAD,  Top  Eleven  Ways  to  Manage  Technical  Risk,  1998,  p.  3- 
3)  The  SOO  is  the  "primary  document  for  translating 
performance  requirements  into  contractual  tasks."  (GSAM, 
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2000,  p.  8-9)  The  SOO  encompasses  top-level  objectives  and 
allows  offerors  greater  flexibility  and  creativity  in 
responding  with  SOWs  that  are  more  detailed.  SOOs  are 
intended  to  not  limit  contractors  by  imposing  restrictive 
specifications . 

Having  offerors  prepare  SOWs  is  intended  to 
reduce  costs  and  time  previously  spent  by  the  Government. 
The  SOO  may  inform  offerors  of  the  Government's  objective 
of  identifying,  assessing,  handling  and  tracking  risk 
areas.  Contractors  may  respond  with  detailed  RMPs  or  risk 
assessment  methodologies  in  the  SOWs  contained  in  their 
proposals.  If  identified  in  Section  M,  offerors'  risk 
management  methodology  contained  in  the  SOW  may  be  used  as 
evaluation  criteria  during  the  SSP . 

Figure  14  illustrates  the  RFP  preparation 

process . 


Figure  14.  RFP  Preparation  Process,  from  (GSAM, 

2000) . 

c.  Risk  Management  Based  Award  Fees  During 
Contract  Administration 

The  process  of  selecting  a  contract  type  is  based 
on  many  factors.  The  contractor's  technical  ability, 
urgency  of  the  program  or  the  type  and  complexity  of  the 
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program  are  all  examples  of  contract  type  decision 
criteria.  Providing  incentives  for  contractors  to  identify 
and  handle  risks  through  Contract  Award  Fees  can  be  a 
valuable  technique  for  the  Government  in  systems 
acquisition  risk  management. 

Award  Fee  determinations  may  be  based  on  analysis 
of  offerors'  SOWs  and  "identification  of  critical  areas  of 
program  risk."  (DAD,  Top  Eleven  Ways  to  Manage  Technical 
Risk,  1998,  p.  3-5)  Moderate  to  high-risk  areas  are 
generally  good  candidates  for  award  fees  because  they 
encourage  the  contractor  to  focus  specific  attention  on 
areas  whose  potential  impact  to  the  program  is  highest. 
Tying  award  fees  to  risk  areas  identified  as  moderate  to 
high  risk  ensures  both  Government  and  contractor  attention 
and  communication.  Award  fee  discussion  between  the 

Government  and  the  Prime  Contractor  should  be  held 
regularly.  Award  fee  discussions  should  be  held  quarterly 
or  even  monthly  to  ensure  continuous  performance  feedback 
for  the  contractor.  This  process  facilitates  open  and 

frequent  communication  between  the  PMO  and  the  Contractor 
and  ensures  that  risk  management  is  a  continuous  process 
that  requires  constant  attention. 

In  order  to  administer  an  award  fee  type 

contract,  contractor  performance  and  award  fee  criteria 
must  be  clearly  articulated  and  measurable.  Award  fee 

periods  are  often  tied  to  specific  events  such  as 

Developmental  Test  and  Evaluation  (DT&E)  events  or 

Operational  Test  and  Evaluation  (OT&E)  events  where  the 

contractor  has  an  opportunity  to  demonstrate  achieved 
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technological  maturity  or  system  integration.  Award  fees 
may  be  incorporated  into  several  contract  types. 

A  Cost  Plus  Award  Fee  (CPAF)  type  contract  is  a 
"cost  reimbursable  contract  that  provides  for  a  fee 
consisting  of  a  base  amount  fixed  at  inception  of  the 
contract  and  an  award  fee  amount  that  the  contractor  may 
earn  in  whole  or  in  part  during  performance."  (Bonneville 
Purchasing  Instructions,  1998,  p.  7-7)  The  program  cost 
estimate  is  determined  during  Government -Contractor 
negotiations.  An  Award  Fee  Plan  (AFP)  is  agreed  to  upon 
inception  of  the  contract.  The  AFP  lays  out  award  fee 
periods  as  well  as  award  criteria. 

The  Government  must  provide  the  Prime  Contractor 
with  frequent  feedback  concerning  contractor  performance 
before  it  can  award,  reduce  or  withhold  an  award  fee.  This 
enables  the  PMO  and  the  contractor  to  communicate 
regularly,  and  openly,  concerning  the  progress  of  the 
program  and  the  status  of  the  program  risk  handling. 
Including  risk  management  as  part  of  award  fee  criteria  is 
a  method  of  transferring  risk  from  the  Government  to  the 
contractor . 

E.  SUMMARY 

This  chapter  first  defined  risk  and  risk  management  in 


the  DoD  environment.  Risk 

is  the  probability 

of  an  event 

occurring  and 

the  likely 

impact  that  event 

has  on 

the 

outcome  of  a 

program. 

Risk  management 

involves 

the 

identification. 

assessment. 

mitigation  and 

finally 

the 

tracking  and  documentation  of  risk  areas  within  a  program. 

Several  categories  of  risk,  or  risk  areas  exist  in  DoD 
acquisition.  A  program's  success  is  based  on  the  system's 
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ability  to  achieve  desired  performance  characteristics 
within  cost  and  schedule  constraints.  Risk  areas,  whether 
technical,  management  or  requirements  can  adversely  affect 
a  program' s  ability  to  comply  with  cost  and  schedule 
constraints  and  still  meet  expected  performance 
characteristics . 

Finally,  this  chapter  discussed  risk  management 
techniques  commonly  used  in  DoD  systems  acquisition.  It 
introduced  T&E,  EVM,  M&S,  CAIV  and  the  contracting  process 
as  areas  with  potential  for  managing  risks. 
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III.  THE  ADVANCED  AMPHIBIOUS  ASSAULT  VEHICLE  (AAAV) 


A.  INTRODUCTION 

This  section  provides  background  information  on  the 
Marine  Corps'  Advanced  Amphibious  Assault  Vehicle  (AAAV) . 
Subsequent  chapters  provide  analysis  of  the  AAAV  program' s 
risk  management  techniques  during  the  Program  Definition 
and  Risk  Reduction  (PDRR)  Acquisition  Phase.  A  Milestone 
III  review  is  planned  for  FY  2007. 

B.  AAAV  HISTORY 

The  Marine  Corps  has  had  an  amphibian  vehicle  since 
1941.  (Lissner  and  Dees,  March  2002)  Its  presence  in  the 
Marine  Corps  arsenal  has  personified  the  Marines' 
amphibious  assault  capability.  The  necessity  for  fighting 
effectively  in  the  littorals  has  continued  to  validate  the 
requirement.  But,  by  1992,  the  currently  fielded  AAV  had 
surpassed  its  planned  10-year  service  life  by  twenty  years. 

Three,  separate  Mission  Area  Analyses  (MAA)  were 
conducted  to  determine  the  AAV' s  mission  effectiveness  and 
suitability.  The  result  was  the  determination  that  the  AAV 
was  deficient  in  mobility  (land  and  water) ,  firepower, 
survivability  and  command  and  control.  (AAAV  DRPM,  Program 
Overview,  October  2002)  The  Center  for  Naval  Analyses 
(CNA)  conducted  an  AOA  to  examine  alternatives  for  the 
Marine  Corps  and  its  amphibious  assault  capability. 
(Lissner  and  Dees,  March  2002) 

The  CNA  presented  thirteen  different  system  solutions 
to  the  Marine  Corps'  needs.  The  alternatives  ranged  from 
amphibian,  swimming  vehicles  to  Landing  Craft,  Air  Cushion 
(LCAC)  loaded  Bradley  Fighting  Vehicles.  Following  the 
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CNA' s  AOA,  the  Marine  Corps  Combat  Development  Command 
(MCCDC)  conducted  a  supplemental  analysis  in  1995.  The 
analysis  yielded  a  "hands  down"  decision  to  pursue  a  "fast  - 
swimming  amphibian."  (Lissner  and  Dees,  March  2002)  The 
criteria  for  analysis  was  based  on  cost,  performance, 
mission  effectiveness  and  total  life  cycle  cost  of  the 
system. 

Following  the  1995  AOA,  the  AAAV  program  entered 
Program  Development  and  Risk  Reduction  (PDRR) .  The  AAAV 
reached  a  Milestone  II  review  in  September  2000.  A 
subsequent  AOA  was  conducted  following  PDRR  to  determine 
the  validity  of  the  proposed  concept  and  its  projected 
costs  (acquisition  and  life  cycle) ,  performance,  and 
mission  effectiveness. 


Figure  15.  The  Advanced  Amphibious  Assault  Vehicle 

(AAAV) ,  from  (FAS,  Military  Analysis  Network, 

October  2002) . 

The  analysis  determined  that  the  AAAV  was,  in  fact, 
the  system  that  the  Marine  Corps  needed  to  overcome 
existing  operational  deficiencies  of  the  AAV.  The  AAAV 
passed  the  Milestone  II  review  in  November  2000  and  entered 
the  System  Development  and  Demonstration  (SDD)  phase.  The 
AAAV  is  currently  in  SDD  as  of  early  2003.  (Lissner  and 


Dees,  March  2002)  (AAAV  DRPM,  Program  Overview,  October 
2002) 

The  AAAV  prime  contractor  is  General  Dynamics 
Amphibious  Systems  (GDAS) .  Its  subcontractors  and 

respective  subsystem  are: 

•  MTU :  Engine 

•  Allison:  Transmission  and  gear  boxes 

•  Honeywell:  Water  jets 

•  Ball:  Antenna 

•  CDC:  Communications  (AAAV  DRPM,  Program  Overview, 

October  2002) 

The  SDD  contract  calls  for  the  production  and  testing 
of  nine  prototypes  prior  to  the  Low  Rate  Initial  Production 
(LRIP)  decision.  As  of  October  2002,  three  prototypes  have 
been  tested  in  varying  terrain  and  under  different 
operational  conditions. 

Prototype  #1  achieved  the  High  Water  Speed  Key 
Performance  Parameter  (KPP)  in  April  2000  by  reaching  a 
maximum  waterborne  speed  of  38  knots.  Prototype  #2  has 
conducted  significant  land  mobility  testing.  The  vehicle 
achieved  Troop  Carrying  and  Land  Speed  KPPs  during  2000. 
The  third  prototype  has  undergone  extensive  land,  water  and 
mobility  developmental  testing  including  Early  Operational 
Assessment  (EOA)  in  2001.  (AAAV  DRPM,  Program  Overview, 
October  2002)  In  the  fall  of  2002,  the  AAAV  performed 
shipboard  launch,  recovery  and  interoperability  testing. 

C.  AAAV  CHARACTERISTICS 

The  Marine  Corps  is  developing  the  AAAV  in  response  to 
noted  operational  deficiencies  in  the  currently  fielded 
Amphibious  Assault  Vehicle  (AAV) .  The  AAAV' s  mission  is  to 
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"provide  high  speed  transport  of  embarked  Marine  Infantry 
from  ships  located  beyond  the  horizon  to  inland 
objectives."  (AAAV  DRPM,  Program  Overview,  October  2002) 
The  AAAV  will  provide  Marine  forces  with  an  enhanced 
capability  to  conduct  ship-to-shore  movement  and  greater 
mobility  and  speed  once  ashore.  The  AAAV' s  range,  speed 
and  enhanced  survivability  will  provide  commanders  with  a 
"Multiple  Options/Late  Decision  Capability."  (AAAV  DRPM, 
Program  Overview,  October  2000)  The  following  table 

illustrates  AAAV' s  improved  capability  over  the  existing 
AAV. 


System 

Function 

AAAV  Requirement 

AAV  Capability 

Water  Speed 

23-29  MPH 

6-8  MPH 

Cross-country 
land  speed 

45  MPH  (keep  pace  with 
main  battle  tank) 

15-20  MPH 

Range  on  water 

65  miles 

45  miles 

Range  on  land 

300  miles 

300  miles 

Troop-carrying 

capacity 

18  combat-equipped 
troops 

18  combat-equipped 
troops 

Survivability 

(1)  4.5mm  round  w/out 
enhanced  armor  plating 

(1)  4.5  mm  round 
with  enhanced  armor 
plating 

NBC  Protection 

Overpressure  System 
(crew  and  embarked 
personnel ) 

None 

Lethality 

Defeat  light  armored 
combat  vehicle  of 

2005-2025  time  frame 
day  and  night  on  the 

move 

Incapable  of 
defeating  light 
armored  combat 
vehicle  with  40mm 
and  .50  cal  weaponry 

Table  1 .  AAAV  and  AAV  Capability  Comparison, 

from  (Military  Analysis  Network/AAAV,  October 
2002  and  General  Dynamics  Land  Systems,  October 

2002)  . 
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The  enhanced  capability  the  AAAV  provides  the  Navy  and 
the  Marine  Corps  is  the  ability  to  conduct  Operational 
Maneuver  From  the  Sea  (OMFTS) .  Littoral  regions  previously 
denying  maneuver  forces  of  options  will  become  maneuver 
areas  and  avenues  of  approach.  The  AAAV  will  extend  the 
Marine  Corps'  "operational  reach."  (AAAV  DRPM,  Program 
Overview,  October  2002) 

D.  AAAV  ACQUISITION  AND  PROCUREMENT 

The  AAAV  is  the  only  ACAT  ID  program  managed  by  the 
Marine  Corps.  The  AAAV  program  office  is  unique  as  the 
Government  and  the  prime  contractor,  GDAS,  share  the  same 
facility  in  Woodbridge,  Virginia. 

The  Marine  Corps  will  buy  one  thousand  and  thirteen 
(1,013)  AAAVs  to  replace  the  AAV  at  a  cost  of  nearly  $7.6 
Billion.  The  AAAV' s  Initial  Operating  Capability  (IOC)  is 
scheduled  for  2008  with  full  fielding  of  the  system 
beginning  in  2012.  (Military  Analysis  Network,  2002,  p.  2) 
Figure  16  depicts  the  AAAV' s  program  schedule  as  of  October 
2002. 


49 


AAAV  PROGRAM  SCHEDULE 

1  July  2002 


Figure  16. 
Direct 


AAA V  Program  Schedule,  from 
Reporting  Program  Manager  (DRPM) , 
2002) . 


(AAAV 

October 
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IV.  THE  ADVANCED  AMPHIBIOUS  ASSAULT  VEHICLE  (AAAV) 
RISK  MANAGEMENT  STRATEGY  DURING  PROGRAM  DEFINITION 
AND  RISK  REDUCTION  ( PDRR) 


A.  INTRODUCTION 

This  chapter  presents  the  research  methodology  this 
author  uses  to  address  the  primary  and  subsidiary  research 
questions.  The  objective  of  this  chapter  is  to  provide 
data  and  information  pertaining  to  the  Advanced  Amphibious 
Assault  Vehicle' s  (AAAV)  Program  Definition  and  Risk 
Reduction  (PDRR)  Phase  risk  management  strategy. 

The  AAAV  Program  Management  Office  (PMO)  is  employing 
several  risk  management  (RM)  techniques  during  the  System 
Development  and  Demonstration  (SDD)  Acquisition  Phase.  The 
next  chapter  analyzes  the  lessons  learned  from  the  PDRR 
risk  management  strategy  and  how  the  lessons  learned  are 
helping  to  shape  the  risk  management  process  during  SDD. 

B.  RESEARCH  METHODOLOGY 

In  order  to  obtain  background  information  to  answer 
the  subsidiary  thesis  questions,  the  author  conducted  a 
thorough  Internet  search  of  Department  of  Defense  (DoD) 
directives,  manuals,  guidelines,  presentations,  handbooks 
and  reports  as  well  as  private  sector  risk  management 
related  sites.  Additionally,  the  author  conducted  a 
literary  search.  The  sources  used  in  the  presentation  of 
the  research  included  newspaper  and  magazine  articles, 
books,  trade  journals  and  other  library  information 
resources.  The  author  visited  the  AAAV  Direct  Reporting 
Program  Management  Office  (DRPM)  in  Woodbridge,  VA  to 
conduct  interviews  with  both  Government  and  Prime 
Contractor  program  leadership  and  observe  risk  management 
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practices  in  use  during  SDD .  The  latter  methodology 
provided  great  insight  into  the  current  DoD  risk  management 
environment . 

The  majority  of  AAAV  specific,  PDRR  and  SDD  Risk 
Management  practices  discussed  in  this  thesis  came  from 
interviews  with  AAAV  PMO  department  heads.  The  author 
interviewed  the  System  Test  Officer,  the  Senior  System 
Engineer,  the  Deputy  Director  of  Program  Planning  and 
Integration  (PP&I)  (also  the  Risk  Management  Coordinator), 
the  System  Chief  Information  Officer  (CIO) ,  and  a  support 
contractor  charged  with  consulting  on  matters  of  risk.  In 
addition  to  Government  program  office  personnel,  the  author 
interviewed  the  General  Dynamics  Amphibious  Systems  (GDAS) 
Risk  Coordinator  (Program  Planning  Integrating  IPT)  , 
Assistant  Risk  Coordinator,  and  a  Quality  Assurance 
Engineer . 

C.  OBJECTIVE  OF  RESEARCH 

The  objective  of  the  research  is  to  collect,  decipher 
and  present  information  and  facts  that  lead  to  the  analysis 
and  discussion  of  DoD  risk  management  practices.  The 
vehicle  for  this  presentation  and  analysis  of  data  was  a 
case  study  of  the  AAAV.  The  primary  and  subsidiary  thesis 
questions  are  re-stated  below: 

1 .  Primary 

•  How  have  the  lessons  learned  from  the  AAAV' s 
Program  Definition  and  Risk  Reduction  (PDRR)  Risk 
Management  Strategy  impacted  the  Program' s  Risk 
Management  Process  during  System  Development  and 
Demonstration  (SDD) ? 

2 .  Subsidiary 

•  What  are  risk  and  risk  management  in  Department 
of  Defense  (DoD)  systems  acquisition? 
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•  What  techniques  can  DoD  use  to  manage  risk  in 
developmental  systems? 

•  What  is  the  AAAV  program? 

•  What  are  the  lessons  learned  from  the  AAAV  PDRR 
Risk  Management  Strategy? 

•  What  risk  management  approaches  has  the  AAAV 
Program  Office  adopted  to  manage  technical  and 
programmatic  risk  during  SDD? 

•  What  conclusions  and  recommendations  can  be  drawn 
from  this  analysis? 

Chapters  II  and  III  answered  the  first  three 
subsidiary  questions.  This  chapter  presents  data  for  the 
purpose  of  addressing  AAAV  risk  management  techniques 
during  PDRR.  Subsequent  chapters  address  the  remaining 
questions  and  conclude  by  answering  the  primary  research 
question  and  providing  conclusions  and  recommendations. 

D.  AAAV  RISK  MANAGEMENT  TECHNIQUES  DURING  PROGRAM 

DEFINITION  AND  RISK  REDUCTION  (PDRR) 

This  section  addresses  the  following  risk  management 
techniques  employed  in  the  Advanced  Amphibious  Assault 
Vehicle  (AAAV)  Program  during  PDRR: 

•  Information  Technology  Tools 

•  The  Joint  Government -General  Dynamics  Risk 
Management  and  Resolution  Process 

•  Contracting 

•  Government  and  Prime  Contractor  Co -Location  and 
the  IPPD  process 

•  Test  and  Evaluation 

1 .  Information  Technology  Tools 

The  AAAV  Team  (Government  and  General  Dynamics 
Amphibious  Systems  (GDAS) )  utilized  robust  information 
technology  tools  to  assist  in  the  management  of  the  program 
during  PDRR.  Information  sharing  within  organizational 
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departments  and  especially  across  functional  areas  was 


crucial  to 

the 

program's  successful 

risk 

management 

approach . 

The 

AAAV  Team  leveraged 

many 

tools  and 

processes . 

This 

section  discusses  the 

Virtual  Design 

Database  (VDD)  and  the  Virtual  Integration  and  Assembly 
(VINTEGRA)  systems . 

a.  Virtual  Design  Database  (VDD) 

The  Virtual  Design  Database  (VDD)  is  a  tool  that 
provides  users  with  a  virtual,  integrated  environment. 
Accessible  to  both  Government  and  GDAS  personnel  at  any 
desk  or  lap  top  computer,  the  VDD  consists  of  user- 
friendly,  windows-based  electronic  documents  sorted  by 
Integrated  Product  Teams  (IPT)  and  aligned  by  the  Work 
Breakdown  Structure  (WBS)  .  The  VDD  "consists  of  various 
distributed  databases  linked  together  via  high  speed 
network  connections."  (Integrated  Digital  Environment 
(IDE),  October  2002)  IPT  areas  include  Modeling  and 
Simulation,  Systems  Engineering,  Logistics,  Mobility 
Products,  Firepower  Products,  Integration  and  Assembly, 
etc.  The  VDD  "provides  AAAV  IPT  members  with  an  on-line, 
real  time,  paperless  communication  system  used  to  logically 
file  and  provide  access  to  AAAV  program  documentation." 
(IDE,  October  2002) 

The  VDD  has  the  following  characteristics: 

•  Makes  data  available  to  all  members,  USMC, 
Government,  and  the  subcontractors  from  a  desktop 
platform 

•  Provides  desktop  3-D  Visualization  and  solid 
model  capability 

•  Stores  data  in  all  electronic  formats 

•  Permits  a  document  author  or  IPT  lead  to  edit  the 
document 
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•  Integrated  e-mail  system 

•  Fully  integrated  with  other  PC  applications. 

(IDE,  October  2002) 

VDD' s  personal  computer  interface  resembles  a 
Work  Breakdown  Structure  (WBS)  format.  Drop-down  windows 
allow  users  to  search  files  by  IPTs.  Risk  Management  is 
facilitated  through  features,  or  capabilities,  built  into 
the  VDD.  The  features  allow  efficient  sorting  of  risks. 
The  VDD  user  is  able  to  select  a  "Risk"  window,  which 
provides  access  to  the  entire  risk  repository  containing 
individual  risk  forms.  VDD  also  has  a  "Help"  function, 
which  assists  unfamiliar  users  in  finding  documents  or 
performing  functions  within  VDD  easily. 

Risk  forms  contain  all  information  pertaining  to 
particular  risks  within  functional  areas  of  the  system. 
The  names  of  the  Risk  Owners  (to  be  discussed  in  the  next 
section) ,  risk  assessments,  status  and  mitigation 
activities  are  contained  on  risk  forms.  Users  may  review 
risks  and  status  of  mitigation  actions  as  well  as  generate 
new  risks.  An  automatic  risk  notification  system  alerts 
other  functional  areas  and  program  leaders  to  emerging 
risks  and  changes  in  risk  status  by  generating  e-mails 
containing  links  to  view  electronic  risk  forms.  Figures 
17,  18,  19  and  20  provide  examples  of  VDD  pages. 
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Figure  17.  Virtual  Design  Database  (VDD) ,  from 

(Rob  Kepner,  EPMC  Brief,  2002)  . 


Electronic  Risk  Forms  allow  VDD  users  to  perform 
a  myriad  of  functions.  One  may  enter  a  new  risk  into  the 
system,  check  the  status  of  current  risk  mitigation  actions 
on  risks  designated  by  specific  risk  numbers,  update  risk 
status,  etc.  Risk  form  editing  is  electronically 
restricted  to  the  designated  Risk  Owners  and  others 
Government  and  contracting  personnel  specifically 
authorized  to  edit  files. 

Figures  18,  19  and  20  illustrate  the  VDD' s  access 
to  risk  forms  and  an  example  of  a  risk  form  as  it  appears 
on  VDD . 
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Figure  18.  VDD  Path  to  Risk  Forms,  from  (Rob 

Kepner,  EPMC  Brief,  2002)  . 


Figure  19. 


VDD  Risk  Form  from  (Rob  Kepner, 
Brief,  2002) . 


EPMC 
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Figure  20.  Organization  of  Risks  on  VDD,  from  (Rob 

Kepner,  EPMC  Brief,  2002)  . 

b.  Virtual  Integration  and  Assembly  (VINTEGRA) 

The  AAAV  Virtual  Integration  and  Assembly  system 
(VINTEGRA)  is  an  engineering  and  manufacturing  tool 
designed  to  support  the  continual  refinement  of  system 
assembly  and  integration.  VINTEGRA  is  a  subset  of  VDD  with 
many  of  the  same  characteristics:  accessibility,  personal 
computer  interface,  automatic  e-mail  notification,  etc. 
VINTEGRA  is  intended  to  capture  and  facilitate  integration 
and  assembly  (I&A)  and  production  data.  The  data  leads  to 
the  identification  of  risks  associated  with  I&A  and 
production.  VINTEGRA,  relying  on  software  called 

ProProcess,  provides  shop  mechanics  with  a  paperless  source 
of  manufacturing  and  assembly  instructions.  VINTEGRA  is 
located  on  AAAV' s  Intranet.  Shop  mechanics  use  a  standard 
Internet  browser,  i.e.,  Internet  Explorer,  to  access 
VINTEGRA' s  drawings  and  assembly  instructions.  VINTEGRA 
may  be  accessed  through  computer  aided  design  (CAD) 
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workstations  in  the  assembly  area  or  on  desktop  computers 
anywhere  on  AAAV' s  network.  "Traditional  'blue-print' 
style  line  drawings  of  the  design  have  been  replaced  by 
computer  images  rendered  in  the  vividness  of  three 
dimensions  (3-D) (Defense  Modeling  and  Simulation  Office, 
1999) 

Multiple  computer  stations  surrounding  the 
vehicle  hulls  undergoing  assembly  are  available  to  the  shop 
mechanics  for  reference.  VINTEGRA  allows  shop  mechanics  to 
electronically  view  the  proper  sequence  and  method  of 
assembly  prior  to  working  on  the  actual  hull.  Mechanics 
control  the  speed  at  which  they  learn  the  self-paced 
assembly  instruction. 

Design  imperfections  and  changes  are  anticipated 
during  prototype  build.  VINTEGRA  is  an  interactive  system. 
Mechanics  may  provide  input  to  engineers  if  they  encounter 
problems  during  assembly  through  an  electronic  problem 
reporting  system. 

VINTEGRA  contains  an  Electronic  Problem 
Resolution  System  (EPRS)  which  allows  for  "real-time 
capture  of  assembly  problems  as  they  are  discovered." 
(Defense  Modeling  and  Simulation  Office,  1999)  If  a 
mechanic  encounters  problems  applying  assembly  instruction, 
then  he  or  she  can  mark  up  the  Computer  Aided  Design  (CAD) 
images  to  illustrate  the  exact  location  and  nature  of  the 
discrepancy.  Because  VINTEGRA  resides  on  AAAV' s  Intranet, 
the  mechanic's  corrections  or  indicated  problem  areas  are 
immediately  brought  to  the  attention  of  engineering  and 
design  teams  via  electronic  dissemination.  The  teams,  in 
turn,  may  make  real-time  changes  to  the  assembly  process  or 
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identify  potential  change  requirements  for  the  next  design 
iteration  via  the  configuration  management  process. 

The  EPRS  form  contains  the  following  major 
subject  headings: 

•  Problem  Identification 

•  Problem  Description 

•  Disposition  of  Short  Term  Fix 

•  Verification  of  Short  Term  Fix 

•  Disposition  of  Long  Term  Fix 

•  Verification  of  Long  Term  Fix 

Shop  mechanics  can  affect  wide  dissemination  of 
the  EPRS  though  VINTEGRA. 

The  AAAV  Team  won  the  1999  Defense  Modeling  and 
Simulation  Office  Award  for  VINTEGRA. 

2 .  AAAV  Team  PDRR  Risk  Management  Process 

The  program's  objective  during  PDRR  is  to  continue  to 
develop  the  design  and  engineering  maturity  of  the  system 
as  well  as  to  identify,  assess  and  handle  risks.  Technical 
risks  tend  to  attract  the  majority  of  attention  during 
PDRR. 

The  AAAV,  Government  program  office  contractually 
mandated  that  GDAS  institute  the  program' s  risk  management 
process  during  PDRR.  The  AAAV  Program  Manager  (PM) 
approved  and  oversaw  the  process.  A  risk  management  plan 
and  its  implementation  by  GDAS  were  second  period  award  fee 
criteria  during  PDRR  based  on  the  Milestone  II  contract 
award.  The  next  section  addresses  Risk  Management  through 
the  contracting  process  in  greater  detail. 


60 


The  AAAV  Team' s  risk  management  process  involved  five 
steps : 

•  Risk  Identification 

•  Risk  Analysis  and  Prioritization 

•  Risk  Planning 

•  Risk  Tracking 

•  Risk  Control  (AAAV  Independent  Risk  Assessment, 

March  2000) 

a.  AAAV  PDRR  Risk  Identification 

The  first  step  in  the  AAAV  risk  management 

process  during  PDRR  was  Risk  Identification.  Any  number  of 
individuals  or  groups  was  empowered  to  identify  risks  and 
initiate  the  management  process.  Individuals,  an  IPT  or 
any  internal  or  external  AAAV  stakeholder  could  identify 
new  or  existing  candidate  risks.  Existing  risks  could  be 
transferred  from  one  IPT  to  another  depending  on  the  nature 
of  the  risk  and  an  IPT' s  expertise.  New  risks  were 

initiated  by  sending  an  e-mail  or  holding  a  conversation 

with  an  IPT  lead  or  the  program  Risk  Management  Coordinator 
(RMC) .  Existing  risks  were  transferred  by  one  IPT  to 

another  or  by  the  Risk  Resolution  Board  (RRB)  .  (GDAS  PDRR 
Risk  Primer,  April  1998) 

The  Joint  Risk  Resolution  Board  (RRB)  was  a  Joint 
DRPM/GDAS  board  designed  to  analyze  and  resolve  risk  issues 
that  required  senior  DRPM  and  GDAS  attention  for 
resolution . 

At  the  RRB,  IPTs  presented  risk  summaries  to  the 
board  members.  Briefs  highlighted  risk  area  trends  and 
emerging  risks  by  IPT.  The  risks  were  categorized  as  High, 
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Moderate  or  Low-level  risks. 


The  AAAV  Team  categorized 


risks  in  the  following  manner: 

•  High  Risk  (Red)  :  Risk  is  most  likely  to  cause 

major  program  impact /disrupt ion .  For  these 

risks,  a  different  approach  may  be  required  to 
successfully  complete  EMD.  Priority  management 
attention  should  be  applied. 

•  Moderate  Risk  (Yellow)  :  Risk  can  cause  some 

program  impact/disruption .  Risk  can  be 

resolvable  during  EMD  by  proper  implementation  of 
mitigation  efforts  that  may  involve  a  different 
approach.  Additional  management  attention  may  be 
needed. 

•  Low  Risk  (Green)  :  Risk  is  at  an  acceptable  level 
with  minimal  or  no  known  impact.  (Independent 
Risk  Assessment  Team  (IRAT) ,  2000) 

Each  risk  was  listed  by  IPT  name  and  by  Risk 
Identification  number  as  seen  on  the  VDD  for  purpose  of 
reference.  (AAAV  Program  Office  Risk  Resolution  Board 
Presentation,  November  1997) 

The  RRB  had  several  objectives.  First  to  provide 
senior  leadership  with  decision  support  information  and 
ensure  cross-functional  area  communication.  Decision 

criteria  included  the  commitment  of  additional  resources, 
acceptance  of  risk,  trade  offs  and  mitigation  courses  of 
action.  Secondly,  to  provide  a  forum  where  decisions  were 
made  jointly  between  GDAS  and  the  Government  using  the  same 
data.  Third,  the  RRB  was  the  appropriate  forum  to  close 
out  risks  nominated  for  this  action.  The  RRB  ensured  that 
the  risk  management  process  was  performed  as  intended. 
Lastly,  the  RRB  intended  to  reduce  risk  processing  and 
handling  cycle  time  to  facilitate  decision-making  at  the 
earliest  possible  opportunity.  (Kepner,  2002) 
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Once  a  candidate  risk  was  identified,  the  issue 
became  an  IPT  or  RRB  agenda  item.  Doctrinally,  IPTs 
reviewed  candidate  risks  weekly  and  existing  risks  bi¬ 
weekly  while  the  RRB  reviewed  program -level  risk 
continually.  The  RRB  attempted  to  meet  formally  on  a 
monthly  basis  as  a  "decision-making  forum."  (GDAS  PDRR  Risk 
Primer,  April  1998)  The  RRB  and  IPTs  reviewed  candidate 
risks  to  determine  one  of  the  following  courses  of  action: 

•  Control  the  risk 

•  Accept  the  risk 

•  Reduce  the  risk 

•  Transfer  the  risk  (Kepner,  2003) 

Determinations  on  candidate  and  existing  risks 

were  made  within  five  working  days  of  identification  and 
initial  discussion.  Candidate  risks  of  minor  severity  were 
classified  as  Action  Items.  IPTs  posted  Action  Items  on 
the  AAAV  Action  Item  database  found  on  the  VDD.  Either  the 
accepting  IPT  or  RRB  entered  risks  on  the  VDD  accompanied 
by  a  mitigation  plan.  Figure  21  illustrates  the  AAAV,  PDRR 
Risk  Management  Process  with  the  Risk  Identification  step 
highlighted  in  gray. 

When  a  risk  was  identified  and  acceptance 
assigned,  the  process  required  further  action.  Formal 
assignment  or  assumption  of  Risk  Ownership  was  required  for 
each  risk  entered  into  VDD. 

The  goal  was  to  resolve  and  close  risks  at  the 
lowest  possible  level.  When  a  risk  was  discovered  during 
PDRR,  the  Government  and  GDAS  counterparts  who  uncovered 
and  introduced  the  risk  became  "Risk  Owners"  for  that  risk. 
They  jointly  assumed  responsibility  for  the  risk  from  its 
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cradle  to  grave.  Risk  Owners  were  identified  by  name  on 
all  risk  forms  on  the  VDD  to  disseminate  the  overall 
responsibility  for  the  particular  risk.  Risk  owners  were 
responsible  for  formally  introducing  new  risks  into  the  VDD 
through  risk  forms. 

Risk  forms  indicated  the  nature  and  severity  of 
the  risk  as  well  as  actions  to  be  taken  to  mitigate  the 
risks . 

The  purpose  of  the  risk  forms  was  to  provide  an 
assessment  of  current  risks  and  estimate  risk  resolution 
requirements  and  timelines.  A  critical  function  of  the 
risk  introduction,  assessment  and  mitigation  process  was  to 
identify  resources  required  to  mitigate  risks.  Risk  Forms 
also  contained  fields  for  the  entry  of  "Estimated  Recovery 
Date"  of  the  risk.  Time  estimates  were  designed  to  allow 
the  program  office  to  analyze  variances  in  actual  versus 
estimated  risk  mitigation  times. 

While  risks  were  jointly  owned  between  GDAS  and 
the  Government,  the  Government  assumed  overall 
responsibility  for  the  risk  and  its  management.  Should  a 
situation  occur  that  a  risk  could  not  be  resolved  and 
closed  at  the  lowest  level  possible,  the  risk  was  elevated 
through  the  IPT  hierarchy  to  the  Joint  Risk  Resolution 
Board  (RRB)  for  resolution. 

The  next  step  in  the  risk  management  process  was 
Risk  Analysis  and  Prioritization. 
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Figure  21.  AAAV  PDRR  Risk  Management  Process:  Risk 

Identification,  from  (GDAS  PDRR  Risk  Primer, 
April  1998) . 
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b.  AAAV  PDRR  Risk  Analysis  and  Prioritization 

The  PDRR  risk  analysis  and  prioritization  process 
began  upon  identification  and  acceptance  of  the  risk. 
Within  five  working  days,  the  owning  IPT  or  RRB  was 
required  to  initiate  a  Risk  Management  Form  in  VDD .  Risk 
forms  in  VDD  were  matched  with  corresponding  Work  Breakdown 
Structure  (WBS)  numbers.  The  IPT  or  RRB  assigned  Keywords 
to  risk  forms.  Keywords  were  "fields  which  enabled  the 
user  to  specify  unique  terminology  associated  with  the  risk 
and  aid  in  the  early  identification  of  trends, 
establishment  of  triggers  and  prioritization  of  constrained 
resources  at  the  IPT  and  system  level."  (GDAS  PDRR  Risk 
Primer,  April  1998)  Keywords  were  classified  as  product, 
practice  or  process: 

•  Product  Keywords:  Names  of  mechanical, 

structural,  software  systems  or  subsystems  unique 

to  AAAV 

•  Practice  Keywords:  Categorized  into  one  of  six 

classes:  Acquisition,  Design,  Facilities, 

Logistics,  Manufacturing  and  producibility  or 

Test  and  Evaluation 

•  Process  Keywords:  Categorized  into  six  classes, 
which  represent  "DoD  Best  Practices":  Design, 
Test,  Production,  Facilities,  Logistics  and 
Management  (GDAS  PDRR  Risk  Primer,  April  1998) 

The  AAAV  RMC  performed  regular  query-based 

searches  of  the  database  to  identify  trends  in  keywords  and 

report  the  findings  to  the  RRB. 

Risk  analysis  included  the  Risk  Owner  preparing  a 
risk  statement  as  a  part  of  the  VDD  Risk  Form.  The  purpose 
of  the  risk  statement  was  to  "quantify  the  cause  of  the 
risk  and  specify  which  requirement  (s)  cannot  be  satisfied 
if  the  risk  is  not  mitigated."  The  risk  consequences  field 


66 


on  VDD  served  to  "quantify  the  effect  of  degraded 
operational  performance,  increased  cost,  decreased 
reliability,  schedule  slip,  etc.  if  the  risk  was  not 
mitigated."  (GDAS  PDRR  Risk  Primer,  April  1998)  The  PDRR 
Risk  Analysis  and  Prioritization  process  is  highlighted  in 
Figure  22 . 

c.  AAAV  PDRR  Risk  Planning 

The  third  step  in  the  PDRR  risk  management 
process  was  Risk  Planning.  Risk  planning  "includes  all 
management  aspects  of  dealing  with  risk  by  choosing  a 
specific  course  of  action  for  mitigation  among  the  several 
alternatives  available."  (GDAS  PDRR  Risk  Primer,  April 
1998)  Risk  planning  included  risk  mitigation  and  risk 
tracking  VDD  entries. 

Risk  mitigation  plans  included  the  following 

elements : 

•  Probability  of  occurrence 

•  Assessed  risk  level 

•  Mitigation  plan  and  history 

•  Estimated  recovery  date 

•  Risk  tracking 

•  Watch  list  inclusion 

•  Date  closed  (GDAS  PDRR  Risk  Primer,  April  1998) 
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Figure  22. 

Analysis 


AAAV  PDRR  Risk  Management  Process:  Risk 
and  Prioritization,  from  (GDAS  PDRR  Risk 
Primer,  April  1998) . 
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The  estimated  recovery  date  was  a  mandatory  entry 
and  reflected  the  IPTs  time  estimate  required  to  mitigate 
the  risk.  When  the  approving  authority  determined  a  risk 
to  be  fully  mitigated  or  no  longer  relevant,  then  the  risk 
was  closed  on  VDD .  The  risk  tracking  entry  allowed  the 
risk  owner  to  select  between  "Off-track"  or  "Monitor 
closely"  depending  on  the  severity  of  the  risk  and  the 
progress  of  the  mitigation  efforts.  "Off-track" 

highlighted  a  risk  that  was  out  of  tolerance  and  "monitor 
closely"  brought  attention  to  a  risk  area  that  was 
approaching  tolerance  limits.  The  categories  allowed 
managers  to  prioritize  risk  areas. 

Included  in  the  mitigation  plan  was  the  assessed 
risk  level  determined  by  the  probability  of  occurrence  and 
the  severity  of  impact  to  the  program. 

Risk  mitigation  plans  included  all  activities  to 
be  conducted  and  resource  estimates  in  order  to  mitigate 
the  risk.  All  mitigation  activities  needed  to  be  completed 
prior  to  the  estimated  recovery  date.  The  risk's  history 
was  updated  each  time  an  IPT  revised  a  risk  form.  The 
mitigation  history  provided  traceability.  (GDAS  PDRR  Risk 
Primer,  April  1998) 

Risks  were  assessed  and  categorized  into  facets 
of  program  impact: 

•  Technical  Performance:  Risk  associated  with  the 
enhancements  of  the  design  to  maximize 
performance 

•  Cost:  Risks  that  impact  budget 

•  Schedule:  Risks  that  impacts  Milestones  and 

Decision  Points 
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Programmatic:  Risks  related  to  resources  (people, 
equipment,  facilities,  funding,  etc.  )  and 
functions  of  the  business 


•  Supportability :  Risks  associated  with  fielding 
and  maintaining  the  current  system  (DRPM  AAA, 
Risk  Mitigation  Planning  Guidance  for  the  Risk 
Owner,  2002) 

The  probability  of  the  risks'  occurrence  and  the 
severity  of  impact  on  one  or  more  of  the  above  mentioned 
categories  determined  the  overall  risk  assessment:  Low, 
Moderate  or  High.  Figure  23  depicts  the  AAAV  PDRR  Risk 
Assessment  matrix. 


Level 

What  is  the  Likelihood 
the  Risk  will  Happen 

a 

Remote 

b 

Unlikely 

c 

Likely 

d 

Highly  Likely 

e 

Near  Certainty 

1  2  3  4  5 


Consequence 


□  HIGH  -  Unacceptable.  Major 
disruption  likely.  Different 
approach  required.  Priority 
management  attention 
required. 


□MODERATE  -  Some 
disruption.  Different 
approach  may  be  required. 
Additional  management 
attention  may  be  needed. 

□  LOW  -  Minimum  impact. 

Minimum  oversight  needed  to 
ensure  risk  remains  low 


LEVEL 

COST 

SCHEDULE 

TECHNICAL 

PERFORMANCE 

PROGRAMMATIC 

SUPPORTABILITY 

1 

Minimal  or 

Minimal  or 

Minimal  or 

Minimal  or 

Minimal  or 

No  Impact 

No  Impact 

No  Impact 

No  Impact 

No  Impact 

2 

<5% 

Additional  Resources 

Acceptable  with  some 

Additional  Resource 

Acceptable  with  some 

Required 

reduction  in  Margin 

Required 

reduction  in  Margin 

3 

5  -  7% 

Minor  Slip  in  Key 

Acceptable,  with  some 

Minor  Slip  in  Key 

Acceptable,  with 

some 

Milestones:  Not  able 

reduction  in  Margin 

Milestones:  Not  able  reduction  in  Margin  1 

to  meet  needed  date  Required 

to  meet  needed  date  Required 

4 

7-10% 

Major  Slip  in  Key 

Acceptable. 

Major  Slip  in  Key 

Acceptable. 

Milestones:  Critical 

No  remaining  Margin 

Milestones:  Critical 

No  remaining  Margin 

Path  impacted 

Path  impacted 

5 

>10% 

Can’t  Achieve  Key 

Unacceptable. 

Can’t  Achieve  Key 

Unacceptable. 

Team  or  major  program 

Team  or  major  program 

Milestone 

Milestone 

Figure  23.  AAAV  PDRR  Risk  Assessment  Matrix,  from 

(GDAS  PDRR  Risk  Primer,  April  1998)  . 
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d.  AAAV  PDRR  Risk  Tracking 

The  RMC  was  largely  responsible  for  risk  tracking 
but  relied  on  IPTs  to  continuously  update  risk  histories 
and  status.  The  RMC  conducted  database  queries  to  search 
for  trends  in  keywords.  The  RMC  compiled  the  query  results 
and  presented  them  to  IPTs  and  the  RRB  for  analysis. 
Analysis  sometimes  included  reallocation  of  resources  to 
address  emerging  risks  or  trends.  The  RMC  served  as 
secretariat  to  the  RRB  in  the  risk  presentation  and 
analysis  process.  Figure  24  illustrates  the  PDRR  Risk 
Tracking  process. 

e.  AAAV  PDRR  Risk  Control 

Risk  control  was  the  day-to-day  management  of 
risks.  (GDAS  PDRR  Risk  Primer,  April  1998)  This  phase  of 
the  risk  management  process  included  the  synthesis  and 
decision-making  and  execution  of  risk  area  courses  of 
action.  The  IPT  or  RRB  responsible  for  the  risk  determined 
one  of  the  following  courses  of  action  in  consonance  with 
the  RMC: 

•  Close  the  risk  and  document  lessons  learned 

•  Evaluate  the  need  for  re-planning  strategies  and 

system- level  workarounds 

•  Invoke  alternative  mitigation  and  contingency 
plans 

•  Continue  to  track  the  risk  and  execute  its 

mitigation 

The  authority  to  make  decisions  on  existing  risks 
was  based  on  the  review  and  approval  authority  matrix  shown 
in  Figure  25. 
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Current  and 
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Risk  Coordinator 
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of  Identification 


Accept  Risk 
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Acceptance 


Action  Item 
Database  More 
Appropriate 


Generate  an 
Action  Item 


Enter  Document 
Information:  WBS 
#/Description,  Title, 
Description  & 
Keywords 


Enter  Risk 
Information:  Log 
Number,  Primary/ 
Secondary  Facets, 
Risk 

Consequences(s) 


Enter  Risk  Mitigation 
Information: 
Estimated  Recovery 
Date,  Assessed  Risk 
Level  and  Probability 
of  Occurrence, 
Include  in  Watch  List, 
Mitigation  Plan  and 
History 
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Attachments  & 
Thumbnail 
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Collect  Data  for 
Tracking  in 
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the  Mitigation/Action 
Plan.  Prioritize 
Mitigation  Rqts 
Against  Available 
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Risk  Mitigation  Plan  to 
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1.  Responsibilities 
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Enter  Approval 
Information: 
Document  Approval 
Status  &  Required 
Document  Approval 
Levels 


Analyze  and 

Compile  Results  of 
Analysis.  Present 

Compile  Data 

to  IPT  and/or  Risk 

Collected  for 

Resolution  Board. 

Tracking.  Look  for 
Trends  and 

- ► 

Re-evaluate 

Resource 

Triggers. 

Allocations.  Decide 

Next  Step. 

Figure  24.  AAAV  PDRR  Risk  Management  Process:  Risk 

Tracking,  from  (GDAS  PDRR  Risk  Primer,  April 

1998) . 
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The  formal  PDRR  risk  management  process  began  on 
conditions  of  a  Cost  Plus  Award  Fee  contract  at  the 
beginning  of  PDRR.  GDAS  developed  the  risk  management 
process  and  the  Government  approved  it.  The  emphasis  of 
the  risk  management  process  was  on  resolving  technical 
risks  during  PDRR. 

The  following  section  discusses  risk  management 
through  the  contracting  process  during  PDRR. 


Origination  Mitigation  Plan  Review  Authorized  to  Closeout 


Source 

and  Approval 

Risk 

High 

A-Level 

IPT 

A-Level 

IPT 

Risk 

Resolution 

Board 

B-Level 

IPT 

A-Level 

IPT 

Risk 

Resolution 

Board 

C-Level 

IPT 

B-Level 

IPT 

Risk 

Resolution 

Board 

D-Level 

IPT 

C-Level 

IPT 

Risk 

Resolution 

Board 

Moderate 

A-Level 

IPT 

A-Level 

IPT 

Risk 

Resolution 

Board 

B-Level 

IPT 

A-Level 

IPT 

A-Level 

IPT 

C-Level 

IPT 

B-Level 

IPT 

B-Level 

IPT 

D-Level 

IPT 

C-Level 

IPT 

C-Level 

IPT 

Low 

A-Level 

IPT 

A-Level 

IPT 

A-Level 

IPT 

B-Level 

IPT 

B-Level 

IPT 

B-Level 

IPT 

C-Level 

IPT 

C-Level 

IPT 

C-Level 

IPT 

D-Level 

IPT 

D-Level 

IPT 

D-Level 

IPT 

Figure 

25. 

AAAV 

PDRR  Risk 

Review 

and  Approval 

Authority  Matrix,  from  (GDAS  PDRR  Risk  Primer, 

April  1998) . 

3 .  Risk  Management  Through  the  Contracting  Process 

This  thesis  introduced  in  Chapter  II  several  methods 
to  reduce  risk  through  the  contracting  process.  Examples 
include  risk-based  Requests  For  Proposal  (RFP) ,  weighting 
of  source  selection  evaluation  criteria.  Statement  of 
Objectives  (S00)  language  and  contractual  incentives.  This 
thesis  presents  the  AAAV  program' s  use  of  contractual 
awards  to  incentivize  risk  management. 
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Following  the  Milestone  I  decision,  the  Government 
awarded  GDAS  a  Cost  Plus  Award  Fee  (CPAF)  contract  for  the 
PDRR  phase.  The  PDRR  Statement  of  Work  (SOW)  required  GDAS 
to  have  a  risk  management  plan.  (Kepner,  2002)  The  PDRR 
Second  Period  Contract  Award  Fee  plan  was  tied  to  GDAS' 
risk  management  performance.  The  DRPM  AAA  established 
performance  criteria  to  evaluate  GDAS'  risk  management 
program  and  supervised  the  process  implementation  during 
PDRR. 

GDAS'  full  award  fee  amount  depended,  in  part,  on  its 
ability  to  implement  an  effective  risk  management  program 
and  manage  risks  during  the  period. 

4.  Government  and  Prime  Contractor  Co-Location 

The  AAAV  DRPM  and  GDAS  have  co- located  at  the  AAAV 
Technology  Center  in  Woodbridge,  VA  and  the  Worth  Avenue 
Technology  Annex  (WATA)  in  Dale  City,  VA  three  miles  south 
of  Woodbridge.  Both  facilities  are  two-story  buildings 
with  the  DRPM  AAA  occupying  the  top  floor  and  GDAS  the 
bottom  floor.  Both  facilities  have  assembly  areas  where 
AAAV  command  mock  ups  and  personnel  variant  prototypes  were 
assembled  during  PDRR  and  continue  to  be  assembled  in  SDD. 
The  AAAV  Team's  co-location  is  truly  unique  in  defense 
acquisitions.  Government  and  Prime  Contractor  co -location 
is  intended  to  enhance  communications  and  reduce  program 
risks  inherent  with  limited  cross-functional  area 
interaction.  One  objective  of  co-location  is  to  facilitate 
the  Integrated  Product  and  Process  Development  (IPPD) 
environment.  Communication  is  crucial  to  successfully 
implementing  an  IPPD  process. 
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a.  Integrated  Product  and  Process  Development 
(IPPD) 

Chapter  II  of  this  thesis  introduced  Integrated 
Product  and  Process  Development  (IPPD)  .  The  DoD  defines 
IPPD  as  "a  management  process  that  integrates  all 
activities  from  product  concept  through  production/f ield 
support,  using  a  multifunctional  team,  to  simultaneously 
optimize  the  product  and  its  manufacturing  and  sustainment 
processes  to  meet  cost  and  performance  objectives." 
(Department  of  the  Navy  Acquisition,  March  2000)  IPPD  is  a 
systems  engineering  process  that  incorporates  the  use  of 
and  interaction  between  multifunctional  teams,  or 
Integrated  Product  Teams  (IPT) .  This  section  provides 
examples  of  IPT  interaction  in  the  IPPD  process  during 
PDRR. 

b.  AAAV  Integrated  Product  Teams  (IPT) 

The  AAAV  program  uses  twenty -eight  IPTs  with 
"membership  representing  every  stakeholder  in  AAAV,  from 
the  Marine  Users,  Government  Civilians,  Industry  (Prime  and 
subcontractors)  ,  up  through  the  Office  of  the  Secretary  of 
Defense."  (Pollution  Prevention:  AAAV,  October  2002)  The 
IPTs  are  involved  in  every  aspect  of  system  development  and 
the  systems  engineering  process.  The  AAAV  IPT  hierarchy  is 
made  up  of  four  levels  of  IPTs  as  indicated  in  Figure  26. 
Figure  27  depicts  the  overall  AAAV  IPT  environment. 
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Figure  26. 


AAAV  IPT  Hierarchy,  from  (Rob  Kepner, 
EPMC  Brief,  2002) . 


AAAV  IPT  ENVIRONMENT 


•  Marine  Amphibious 
Vehicle  Crewman  and 
Maintainers  are  Part  of 
Every  Integrated  Product 
Team 

•  Along  With  the  Prime 
Contractor,  Subcontractors, 
and  Government  Engineers 
and  Logisticians 


•  Every  Design  Decision 
is  Made  Considering  its 
Effect  on  Combat 
Effectiveness, 
Maintainability,  and 
Operations  and  Support 
Costs 


Analysis  / 

Simulation  /  Weight 


Development 
Software^  Test 

Process  Control 


Turret  &  Armament 
Fire  Control 


P&D 

Power  Controls 


Figure  27.  AAAV  IPT  Environment,  from  (DRPM  AAA 

IPT  Brief,  October  2001)  . 


76 


Co-location  allows  IPTs  to  work  in  continual, 
close  proximity  with  Government,  user  and  contractor 
counterparts.  Numerous  risk  areas  emerge  during  prototype 
development  and  production.  The  opportunity  for  IPT 
members  to  work  together  and  communicate  across  functional 
area  lines  on  a  daily,  regular  basis  is  critical  to 
capturing  and  managing  risks. 

In  the  area  of  Environmental  Safety  and  Health 
(ESH)  ,  each  IPT  has  a  member  who  provides  expertise  and 
representation  for  ESH  issues.  ESH  issues  play  a 
significant  role  in  the  AAAV  development.  ESH  issues 
represent  risk  to  the  program  in  the  form  of  large  disposal 
costs,  potential,  adverse  environmental  impact  and  safety 
of  use  concerns.  The  following  section  uses  ESH  to 
illustrate  IPT  interaction  in  the  development  of  a  complex 
system. 

c.  AAAV  Environmental  Safety  and  Health  Program 

The  National  Environmental  Protection  Act  of  1969 
(NEPA)  mandates  Government  consideration  of  environmental 
impacts  imparted  by  system  development,  fielding  and 
disposal.  In  Chapter  5,  the  DoD  5000. 2 -R  states: 

The  PM  shall  evaluate  and  manage  the  selection, 
use,  and  disposal  of  hazardous  materials 
consistent  with  ESOH  regulatory  requirements  and 
program  cost,  schedule  and  performance  goals. 

(DoD  5000. 2-R,  April  2002) 

The  potential  environmental  impact  of  an 
acquisition  system  can  create  risks  to  the  program.  Risk 
areas  include  cost  (development,  life  cycle  and  disposal)  , 
schedule  and  performance  risks.  The  effective  management 
of  ESH  issues  in  a  developmental  weapon  system  is  critical. 
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ESH  representation  in  the  IPPD  process  is  vital  to 
addressing  potential  risks  for  trade  off  analyses. 

Throughout  the  AAAV' s  early  program  definition 
and  continuing  through  SDD,  ESH  issues  have  impacted  the 
system' s  design  and  planned  production,  fielding  and 
disposal.  ESH  factors  can  present  significant  risk  to  the 
program  should  the  system  fail  to  meet  EPA  standards  or 
generate  excessive  costs  attributable  to  minimizing 
environmental  impact  during  operation  or  disposal.  The 
AAAV  Team  has  worked  together  to  ensure  the  optimal  balance 
exists  between  the  AAAV' s  performance  and  System  Lifecycle 
Costs . 

AAAV  IPTs  each  included  ESH  representatives 
during  PDRR.  The  representatives  were  tasked  with 
including  ESH  considerations  during  the  system' s  design  and 
development  trade  off  processes.  The  intent  was  to  ensure 
continual  consideration  of  potential  environmental  risk 
impacts  to  the  program  throughout  the  design  and  prototype 
manufacturing  process. 

5.  Test  and  Evaluation  (T&E) 

During  PDRR,  the  AAAV  program  office  produced  three, 
fully  functional  prototypes  for  Developmental  Testing  (DT) 
and  limited  Operational  Testing  (OT) :  PI,  P2  and  P3. 

The  DRPM  AAA  conducted  extensive  DT  during  PDRR.  P2 
conducted  4854  miles  of  land  mobility  testing:  equivalent 
to  nine  vehicle  years.  (DRPM  AAA,  November  2002)  .  P2  was 
also  used  in  an  Early  Operational  Assessment  (EOA) . 

An  EOA  is  a  type  of  test  "conducted  prior  to,  or  in 
support  of  prototype  testing."  (Test  and  Evaluation 
Management  Guide,  November  2001)  A  combination  of  AAAV  PMO 
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Marines  and  5th  to  95th  percentile  users  manned  the  vehicle 
during  the  assessment  held  in  29  Palms,  California  in 
October  2001.  The  purpose  of  the  EOA  was  to  identify 
necessary  design  and  configuration  changes  to  the  vehicle. 
The  PMO  also  conducted  a  gunnery  EOA  to  assess  the  AAAV' s 
weapon  and  fire  control  systems.  The  EOA  enabled  the  AAAV 
program  to  solicit  feedback  on  the  system  from  users  at  an 
early  stage  in  the  system's  development  and  testing. 

PI  and  P3  were  used  primarily  for  water  mobility 
testing.  P3  also  participated  in  weapon  station  and  land 
mobility  test  events. 

PI  and  P3  testing  included  vehicle  transition  from 
water  to  dry  land  and  vice  versa.  The  prototypes  were 
tested  in  the  high  water  mobility  mode  to  assess  the 
vehicle's  ability  to  achieve  threshold  high  water  speed 
requirements.  The  program  office  conducted  informal  user 
juries  with  the  five  Marines  who  performed  as  DT  vehicle 
crews.  The  user  juries  were  encouraged  to  provide 
continuous  feedback  to  system  design  engineers  throughout 
testing . 

In  addition  to  land  and  water  mobility  testing,  the 
AAAV  prototypes  underwent  communication  testing  (with  use 
of  a  vehicle  mock-up) ,  firepower  testing,  survivability 
testing,  habitability  testing  and  lifecycle  support 
testing. 

Extensive  survivability  testing  included  mock-up  and 
prototype  test  activities  to  evaluate  the  system' s  armor 
protection  and  crew  survivability.  In  addition,  the  PMO 
placed  significant  effort  in  developing  and  testing  an  on - 
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board  automatic  fire  suppression  system  to  increase  crew 
survivability . 

Life  cycle  support  testing  consisted  of  logistics 
demonstrations,  maintainability  demonstration,  mean  time  to 
repair  (MTTR)  demonstration,  interactive  electronic 
technical  manual  validation  (IETM)  and  human  factors 
engineering  testing.  (DRPM  AAA,  November  2002) 


Figure  28.  AAAV  Survivability  Testing,  from  (AAAV 
Developmental  Test  Brief,  November  2002)  . 

E.  SUMMARY 

During  PDRR  the  AAAV  Team  worked  to  identify,  assess, 
mitigate  and  track  technical  and  programmatic  risks.  The 
AAAV  Team  employed  several  tools  and  techniques  to  manage 
risk . 

General  Dynamics  Amphibious  Systems  (GDAS)  developed 
information  technology  tools  to  assist  in  the  risk 
management  process  during  PDRR.  The  Virtual  Design 
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Database  (VDD)  was  an  Intranet  tool  designed  to  facilitate 
communication  within  and  across  both  Government  and 
contractor  functional  lines.  VDD  contained  a  "Risk" 
section,  which  enabled  IPT  members  to  initiate,  track, 
update,  and  disseminate  risk  items  throughout  the  AAAV 
program  office. 

Virtual  Integration  and  Assembly  (VINTEGRA)  is  an 
engineering  and  manufacturing  tool  that  provides  shop 
mechanics  with  3-D  views  of  system  assembly  instructions 
and  processes.  VINTEGRA  has  an  Electronic  Problem 
Reporting  System  (EPRS)  that  allows  shop  mechanics  to 
provide  feedback  to  system  designers  and  engineers  on  the 
assembly  process  and  component  design.  VINTEGRA  provides 
real-time  updates  to  technical  data  packages  (TDP)  shared 
by  GDAS  and  the  DRPM  AAA. 

The  AAAV  Team  encouraged  system  risk  input  at  all 
levels.  Individuals  who  identified  risks  became  "Risk 
Owners".  Government  and  contractor  counterparts  shared 
risk  ownership. 

Risk  Owners  could  initiate,  edit  and  communicate  all 
subsequent  inputs  or  changes  to  risk  areas  Online  through 
the  use  of  Risk  Forms  found  on  the  VDD. 

Risks  are  ideally  resolved  at  the  lowest  possible 
level  in  the  IPT  hierarchy.  Those  risks  deemed  significant 
program  risks  or  those  incapable  of  resolution  at  lower  IPT 
levels  were  elevated  to  a  Joint  Risk  Resolution  Board 
(RRB)  .  The  Government  and  GDAS  leadership  comprised  the 
RRB.  The  purpose  of  the  RRB  was  to  provide  the  program 
leadership  with  decision  support  information  and  clearly 
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communicate  risks  across  functional  lines  at  the 

Integrating  IPT  level. 

The  PDRR  contract  was  a  Cost  Plus  Award  Fee  (CPAF) 
type.  The  award  fee  assessment  included  the  evaluation  of 
GDAS'  risk  management  plan  and  its  implementation.  The 

Government  used  award  fees  as  an  incentive  for  the  Prime 
Contractor  to  proactively  manage  risks. 

The  AAAV  Team  is  co-located  in  Woodbridge,  Virginia  in 
the  AAAV  Technology  Center  and  the  Worth  Avenue  Technology 
Annex;  the  facilities  are  approximately  three  miles  apart. 
The  DRPM  AAA  members  and  GDAS  employees  work  in  the  same 
buildings.  The  principal  purpose  of  co-location  is  to 
encourage  and  facilitate  continual  communication  between 
Government  and  contractor  personnel  and  across  functional 
lines.  Co-location  is  consistent  with  the  principles  of 
the  systems  engineering  and  IPPD  processes. 

Co-location  enables  IPTs  to  meet  regularly  without 
significant  travel  requirements  or  disruption  of  other 
responsibilities  in  the  program  offices.  This  thesis 
illustrates  the  program  use  of  the  IPPD  process  through  a 
case  study  involving  environmental  safety  and  occupational 
health  issues. 

Three  AAAV  prototypes  underwent  extensive 
developmental  and  early  operational  testing  during  PDRR. 
The  purpose  of  the  PDRR  test  plan  was  to  identify 
performance  risk  areas  and  solicit  user  input.  The  PDRR 
test  plan  included  logistics  and  life  cycle  testing. 

The  next  chapter  provides  analysis  of  the  PDRR  risk 
management  plan.  This  thesis  discusses  lessons  learned 
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from  the  PDRR  phase  and  analyzes  how  those  lessons  learned 
have  helped  shape  the  AAAV  risk  management  strategy  during 
SDD . 
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V.  ANALYSIS  OF  THE  ADVANCED  AMPHIBIOUS  ASSAULT 
VEHICLE  (AAAV)  RISK  MANAGEMENT  PROCESS  DURING 
PROGRAM  DEFINITION  AND  RISK  REDUCTION  (PDRR) 


A.  INTRODUCTION 

This  chapter  analyzes  data  presented  in  the  previous 
chapter.  In  Chapter  IV,  this  thesis  presented  elements  of 
the  Advanced  Amphibious  Assault  Vehicle  (AAAV)  risk 
management  program  during  the  Program  Definition  and  Risk 
Reduction  (PDRR)  acquisition  phase.  The  data  was 

categorized  into  five  areas  of  research: 

•  AAAV  Information  Technology  Tools 

•  The  Joint  Government -General  Dynamics  Risk 

Management  and  Resolution  Process 

•  Managing  Risk  Through  the  contracting  process 

•  Government  and  Prime  Contractor  Co -location  and 
the  IPPD  process 

•  Test  and  Evaluation  (T&E) 

B.  RESEARCH  METHODOLOGY 

This  author' s  research  methodology  involved  extensive 
Internet  and  literary  searches.  There  are  abundant 

resources  available  to  research  the  Department  of  Defense 
(DoD)  risk  management  environment  and  practices.  However, 
with  the  cancellation  of  the  DoD  5000  Series  of  documents, 
further  research  is  necessary  in  the  future  to  ascertain 
the  revised  risk  management  directives  and  methodologies 
used  in  DoD  acquisitions.  As  this  thesis  is  a  case  study 
of  a  developmental  program  that  began  under  the  DoD  5000.1 
and  DoD  5000. 2-R  directives,  the  associated  DoD  risk 

management  practices  are  evident  throughout  the  program' s 
history  from  PDRR  through  System  Development  and 

Demonstration  (SDD)  .  What  will  be  of  interest  is  how  the 
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revised  acquisition  guidelines  impact  current  and  future 
risk  management  processes  in  developmental  weapon  systems 
including  the  Advanced  Amphibious  Assault  Vehicle  (AAAV) . 

This  author  visited  the  AAAV  program  office  in 
Northern  Virginia.  The  ability  to  observe  program 

operating  procedures  and  interview  both  Government  and 
contractor  personnel  was  invaluable  to  this  thesis. 

C.  OBJECTIVE  OF  RESEARCH 

The  purpose  of  this  chapter  is  to  analyze  the  risk 

management  techniques  the  AAAV  Team  used  during  PDRR. 
Based  on  the  analysis  of  the  PDRR  risk  management  process 
and  techniques,  this  thesis  intends  to  tie  correlations 

between  lessons  learned  from  PDRR  to  the  risk  management 
process  currently  being  used  in  SDD . 

D.  ANALYSIS  OF  RISK  MANAGEMENT  TECHNIQUES  USED  DURING 

PROGRAM  DEFINITION  AND  RISK  REDUCTION  (PDRR) 

The  purpose  of  this  section  is  to  analyze  the  risk 

management  techniques  and  processes  the  AAAV  Team  used 
during  PDRR.  This  chapter  addresses  risk  management 
process  changes  from  PDRR  to  SDD,  the  system's  current 

acquisition  phase.  This  chapter  analyzes  data  in  the 
following  sequence: 

•  AAAV  Information  Technology  Tools 

•  The  Joint  Government -General  Dynamics  Risk 

Management  and  Resolution  Process 

•  Managing  Risk  Through  the  Contracting  Process 

•  Government  and  Prime  Contractor  Co -location  and 
the  IPPD  Process 

•  Test  and  Evaluation 


86 


1 .  Information  Technology  Tools 

The  AAAV  Team's  use  of  information  technology  (IT) 
tools  during  PDRR  was  invaluable.  The  joint  Government  - 
General  Dynamics  Amphibious  Systems  (GDAS)  IPPD  process  was 
enhanced  by  the  use  of  both  the  Virtual  Design  Database 
(VDD)  and  the  Virtual  Integration  and  Assembly  (VINTEGRA) . 
Recognizing  the  "communication  multiplier"  effect  of  tools 
like  VDD,  AAAV  is  developing  a  similar,  upgraded 
application.  (Kepner,  2002)  The  following  sections  discuss 
the  lessons  learned  and  SDD  perspectives  of  both  VDD  and 
VINTEGRA. 

a.  Analysis  of  the  Virtual  Design  Database 

The  Virtual  Design  Database  (VDD)  provided  the 
AAAV  program  office  with  a  top-level  view  of  the  overall 
program.  In  particular,  VDD  provided  a  central  repository 
for  risk  forms  containing  identification,  assessment, 
mitigation  planning  and  flexible  documentation.  VDD's 
organization  matched  the  system' s  Work  Breakdown  Structure 
(WBS)  from  which  the  IPT  organization  was  aligned.  VDD  was 
a  logical,  Lotus  Notes-based  application  that  facilitated 
involvement  at  all  levels  in  the  risk  management  process. 

VDD's  built-in  help  function  facilitated  risk 
data  entry.  Significant  emphasis  was  placed  on  making  VDD 
user-friendly  in  order  to  avoid  discouraging  IPT  members 
from  using  the  system.  VDD's  automatic,  interactive  risk 
notification  system  ensured  widest  dissemination  of  risk 
forms  and  updates  throughout  the  program  offices.  The 
automatic  notification  system  was  interactive  because  it 
required  the  determination  of  candidate  risks  to  be  made 
within  five  days  of  identification  and  electronic 
notification.  IPT  leads  and  program  leadership  were 
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involved  in  risk  identification,  mitigation  plans  and 
tracking  from  the  outset  of  new  risks.  VDD  helped  to 
reduce  the  occurrence  of  poor  communication  within  the 
program  office.  VDD  applications  helped  to  implement  the 
risk  management  process. 

Risks  identified,  assessed  and  entered  into  the 
VDD  were  required  to  contain  mitigation  plans.  The  plans 
included  estimated  recovery  dates  and  anticipated  resources 
required  to  mitigate  the  risk.  Mitigation  activities  were 
tracked  through  the  VDD  as  described  earlier  in  this  and 
the  previous  chapter. 

To  manage  technical  risks,  the  AAAV  Team 
conducted  extensive  Modeling  and  Simulation  (M&S)  ,  Cost  as 
an  Independent  Variable  (CAIV)  trade-off  analyses,  and 
advanced  technology  demonstrators  (ATD) .  The  purpose  of 
the  ATDs  was  to  evaluate  the  military  utility  and 
effectiveness  of  advanced  technology  concepts  and  to 
prepare  to  transition  capabilities  into  the  acquisition 
cycle . 

The  AAAV  PMO  used  ATDs  and  M&S  extensively  during 
PDRR  to  manage  technical  risks  and  perform  trade  off 
studies.  The  AAAV  PMO  built  a  4/5 -size  hydrodynamic  test 
rig  to  prove  the  planning  craft  technology  as  well  as  an 
automotive  test  rig  to  prove  the  vehicle' s  retractable 
suspension  and  lightweight  track  technology.  (Kepner,  2003) 
In  M&S,  the  PMO  used  the  NATO  Reference  Mobility  Model 
(NRMM)  using  AAAV  vehicle  characteristic  inputs  (approach 
angle,  departure  angle,  weight,  height,  etc.)  to  simulate 
vehicle  terrain  handling  and  mobility  characteristics. 
Both  ATD  and  M&S  activities  were  intended  to  greatly 


88 


mitigate  technical  risks  at  the  outset  of  PDRR  by 
identifying  key  system  characteristics  and  identifying 
potential  risk  areas. 

The  result  of  the  risk  management  efforts  during 
PDRR  was  the  production  of  three  fully  functional  AAAV 
prototypes.  (Kepner,  EPMC  Brief)  VDD  was  a  way  of 
communicating  the  results  of  trade-off  analyses  and  other 
risk  mitigation  activities. 

The  VDD  was  an  effective  risk  communication  tool 
during  PDRR.  One  important  characteristic  of  VDD  was  the 
"write  once,  read  many"  quality  of  the  application. 
(Kepner,  2003)  "Write  once,  read  man"  illustrates  VDD' s 
utility  as  a  communications  multiplier  within  the  program 
office.  However,  GDAS  and  AAAV  Government  personnel  feel 
that  the  system  was  out-dated  and  incapable  of  being 
effective  during  SDD .  Some  program  office  personnel 
considered  VDD  a  risk  repository  that  eventually  became 
saturated  with  data.  (Rose,  2002)  VDD  lacked  functional 
aspects  that  the  program  office  feels  are  important  during 
SDD:  trend  analysis,  metric  reporting  capabilities,  and 
analysis  of  variance  (ANOVA)  .  GDAS  felt  that  VDD  risk 
mitigation  plans  were  stand-alone  documents,  not  fully 
integrated  into  the  overall  program  management.  The 
Independent  Risk  Assessment  conducted  prior  to  Milestone  II 
found  that  VDD  was  "more  of  a  status  report  of  activities 
or  a  list  of  planned  events  or  meetings"  rather  than  useful 
mitigation  plans.  (IRAT,  September  2000) 

Subcontractors  had  limited  access  to  VDD.  High 
and  moderate  risks  owned  by  sub-contractors  were 
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incorporated  into  VDD .  As  the  AAAV  program  grew,  the  need 
for  an  expanded  system  became  evident. 

In  SDD,  the  AAAV  is  developing  an  upgraded 
application  that  will  replace  VDD.  The  new  system.  Life 
Cycle  Information  System  (LCIS) ,  is  a  web-based  application 
that  will  allow  all  Government,  Prime  and  Sub-contractors 
to  access  the  database  from  AAAV  desktop  machines  as  well 
as  remotely  through  standard  Internet  browsers.  Another 
impetus  behind  the  development  of  LCIS  is  that  the  AAAV 
program  has  been  designated  as  a  Program  Management  Office 
of  Life  Cycle  Support  (PMOLCS) .  A  PMOLCS  designation 
directs  that  a  PMO  maintain  responsibility  for  the  system 
from  "cradle  to  grave."  (Kepner,  2003)  LCIS  will  be  better 
equipped  to  support  this  function  than  was  VDD. 

LCIS  training  is  scheduled  to  begin  in  January 
2003.  The  AAAV  Team  recognizes  the  importance  of 
incorporating  sub -contractors  in  the  risk  management 
process.  AAAV  intends  to  fully  train  sub-contractors  on 
LCIS  and  include  them  in  the  risk  management  process  by 
offering  this  web-based  application. 

GDAS,  through  their  sub -contractor ,  Computer 
Systems  Corporation  (CSC) ,  is  developing  LCIS  to  become  a 
"customized  document  management  system."  (Rose,  2002)  LCIS 
will  be  capable  of  conducting  ANOVA  when  analyzing 
mitigation  efforts.  LCIS  will  track  projected  resource 
outlays  versus  predicted  outlays;  determine  the 
effectiveness  of  the  mitigation  plan;  and  alert  program 
leadership  to  cost,  schedule  or  programmatic  impacts.  The 
AAAV  Government  team  feels  that  LCIS  will  be  able  to 
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support  a  life  cycle  support  system  that  emphasizes 
program-wide  impact  of  risks. 


b.  Analysis  of  Virtual  Integration  and  Assembly 
(VINTEGRA) 

The  Defense  Modeling  and  Simulation  Office 
awarded  the  AAAV  Program  Management  Office  (PMO)  the  1999 
award  for  excellence  in  Modeling  and  Simulation.  The 
benefits  gained  from  VINTEGRA  during  PDRR  are  already 
evident  in  SDD  and  anticipated  during  subsequent  production 
activities . 

Refining  engineering  assembly  knowledge  benefited 
PDRR  efforts  by  accomplishing  "scope  commonly  done  during 
Engineering  and  Manufacturing  Development  (E&MD)  (now  SDD) 
phase."  (Defense  Modeling  and  Simulation  Office,  1999) 
Furthermore,  the  "information  captured  using  VINTEGRA  in 
PDRR,  significant  data  for  production  line  analyses  will  be 
collected  before  the  conclusion  of  PDRR."  (Defense  Modeling 
and  Simulation  Office,  1999)  The  data  collected  during 
prototype  builds  in  PDRR  will  be  applied  to  SDD  vehicles. 

The  SDD  prototype  vehicle  assembly  process 
benefited  from  the  use  of  VINTEGRA  during  PDRR.  The 
assembly  process  has  been  refined  and  the  Government  and 
contractor  have  identified  and  corrected  many  manufacturing 
and  produceability  issues  during  the  first  three,  PDRR 
prototype  builds.  Both  the  Government  PMO  and  GDAS 
consider  production  to  be  low  risk  partly  due  to  VINTEGRA 
and  the  prototype  assembly  process. 

The  process  sheets  in  VINTEGRA  are  continually 
refined  and  updated.  In  essence,  the  manufacturing  process 
during  PDRR  and  for  prototype  builds  was  a  prototype  of  the 
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eventual  manufacturing  process.  The  goal  was  to  reduce 
risks  associated  with  system  manufacturability,  such  as 
schedule  and  cost  risks. 

Shop  mechanics'  familiarity  with  vehicle  process 
sheets  and  the  Internet-based  browser  and  hyperlinks  in 
VINTEGRA  is  expected  to  result  in  a  learning  curve  effect. 
The  benefits  from  the  learning  curve  will  be  realized  in 
the  form  of  reduced  vehicle  build-time,  or  cycle  time,  and 
improved  quality  in  manufacturing.  The  result  will  be 
reduced  risk  of  missing  system  delivery  dates  and  costly 
re-work  of  the  system  at  the  end  of  the  production  process. 

Additionally,  the  re-use  of  VINTEGRA' s  three 
dimensional  drawings  and  process  sheets  are  expected  to 
avoid  costs  normally  incurred  in  the  development  of 

Interactive  Electronic  Technical  Manuals  (IETM) .  The 
electronic  process  sheet  drawings  have  been  validated  and 
will  be  used  in  the  IETM  development. 

Electronic  tools  like  VINTEGRA  show  potential  for 
"shortening  acquisition  lead  time  and  meeting  war  fighter 
needs  faster,  better,  and  cheaper,  with  the  consequence  of 
lower  risk  to  the  program."  (Defense  Modeling  and 

Simulation  Office,  1999)  This  will  enable  the  Marine  Corps 
to  field  a  higher  quality  system  to  the  war  fighter  at  a 
lower  design  to  unit  production  cost  (DTUPC) .  VINTEGRA  has 
enabled  the  AAAV' s  Integrated  Product  and  Process 
Development  (IPPD)  to  anticipate  system  changes  early  in 
the  vehicle's  prototype  design  and  manufacture. 
Implementation  of  necessary  design  changes  early  in  the 
system' s  development  will  reduce  technical  and  integration 
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risks  and  the  risk  of  increasing  DTUPC  and  delayed  system 
delivery . 

Realizing  the  success  earned  through  the  use  of 
information  technology  (IT)  tools  during  PDRR  as  well  as 
the  need  to  expand  existing  capabilities,  the  AAAV  program 
continues  to  innovate  acquisitions  through  its  creative  use 
of  IT  applications  during  SDD . 

The  Department  of  the  Navy  (DoN)  Electronic 
Business  (eBusiness)  Operations  Office  sponsored  the  DRPM 
AAA  office  to  conduct  a  pilot  program  to  pursue  expanding 
VINTEGRA  and,  eventually,  LCIS  connectivity  to  remote 
sites.  Sites  such  as  remote  test  facilities  did  not 
previously  have  access  to  the  information  and  communication 
capabilities  captured  by  the  AAAV' s  IT  applications  during 
PDRR.  AAAV' s  creative  IT  innovations  will  continue  to 
expand  benefits  in  managing  program  risks  during  SDD  by 
creating  "virtual,  integrated  environments."  (Hepler,  2002) 

2 .  Analysis  of  AAAV  PDRR  Risk  Management  Process 

As  discussed  in  the  previous  section,  the  AAAV  PDRR 
risk  management  process  relied  heavily  on  the  VDD  to 
initiate,  assess,  communicate  and  track  risks.  However, 
VDD  was  only  a  tool  to  help  execute  the  formal  risk 
management  process. 

The  PDRR  risk  management  process  involved  five  steps: 

•  Risk  Identification 

•  Risk  Analysis  and  Prioritization 

•  Risk  Planning 

•  Risk  Tracking 

•  Risk  Control 
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This  section  analyzes  the  PDRR  risk  management  process 
and  discusses  the  lessons  learned.  The  lessons  learned 
from  the  PDRR  risk  management  process  helped  to  shape  the 
proposed  Government/GDAS  risk  management  processes  for  SDD . 
At  the  time  this  thesis  is  being  written,  the  formal  SDD 
process  is  not  yet  implemented.  This  thesis  briefly 
discusses  the  interim  risk  management  process.  The  impetus 
of  the  risk  management  process  is  transitioning  from 
technical  to  system-level  integration  risks.  However,  the 
relevance  of  this  analysis  is  in  the  discussion  of  what 
worked  and  what  did  not  work  during  PDRR  and  how  those 
lessons  learned  are  put  to  use  during  SDD. 

a.  AAAV  PDRR  Risk  Management  Process 

The  AAAV  PDRR  risk  management  process  was 

effective.  The  primary  goal  was  to  identify,  assess,  and 
mitigate  technical  risk  areas  to  support  decision-making 
and  design  trade-offs.  By  focusing  on  technical  risk 
management,  the  AAAV  program  was  able  to  conduct  CAIV 

analyses  and  perform  modeling  and  simulation  activities  to 
determine  the  most  effective  design  characteristics  with 
respect  to  cost,  schedule  and  performance.  Key  elements  to 
the  PDRR  risk  management  process  were  ease  of 
implementation  via  the  VDD  and  effective  communications  of 
the  process. 

The  AAAV  Team  communicated  the  formal  risk 

management  plan  through  the  IPT  levels  via  a  Risk  Primer. 
The  Risk  Primer  provided  risk  management  process 
instructions  intended  for  use  as  a  desktop  reference  for 
risk  management.  The  Risk  Primer  was  an  easy  to  use, 

procedural  nineteen-page  document.  The  intent  was  to 
introduce  and  include  risk  management  in  the  normal 
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operations  of  the  program  office,  not  to  bog  IPT  members 
down  in  formal  methodology  details.  The  primer  provided 
instructions  on  identification,  analysis,  planning, 
tracking  and  control  processes  and  use  of  the  VDD. 

The  lesson  learned  was  that  a  simplif  ied  risk 
management  methodology  education  process  is  crucial  to  the 
successful  implementation  of  a  risk  management  plan.  The 
process  and  education,  or  training,  must  extend  to  the 
subcontractors  as  well  as  the  prime  contractor  and 
Government  personnel.  As  the  program  transitions  from 
development  to  production  preparation,  it  becomes 
especially  important  to  maintain  a  formal  risk  management 
process  with  all  program  stakeholders.  Once  the  SDD 
process  is  formalized,  the  program  intends  to  incorporate  a 
risk  primer  as  part  of  the  process  training. 

The  methodology  the  AAAV  Team  used  to  manage 
risks  during  PDRR  was  sound.  Because  the  majority  of  risks 
during  PDRR  were  technical  risks,  the  DRPM  AAA  RMC  was  a 
systems  engineer  with  a  part-time  risk  focus.  The  RMC  was 
well  equipped  to  coordinate  and  manage  risk  areas  because 
of  the  integral  role  that  he  played  during  the  acquisition 
phase.  As  the  AAAV  program  transitioned  from  PDRR  to  SDD, 
the  risk  management  coordination  also  transitioned. 

Additionally,  shortfalls  and  limitations  of  the 
VDD  are  currently  being  addressed  through  the  development 
of  LCIS.  Currently,  the  AAAV  Team  uses  shared  drives 
available  on  all  desktop  computers  connected  to  the 
program's  Intranet.  IPTs  use  standardized  risk  forms  to 
initiate  the  risk  management  process.  The  risk  forms  used 
in  SDD  differ  from  those  used  in  PDRR. 
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The  SDD  risk  forms  are  Microsoft  Word  documents 
that  capture  the  risk  assessment  and  other  pertinent 
information,  as  a  placeholder  pending  the  full 
functionality  of  LCIS.  Figure  30  illustrates  the  current 
risk  form  the  AAAV  Team  uses  during  SDD.  Risk  forms 
contain  information  on  the  following: 

•  Risk  Ownership 

•  Risk  Title  and  description 

•  Risk  Assessment 

•  Risk  Mitigation  Plan  and  status 

The  current  risk  forms  and  their  location  on  the 
DRPM  AAA/GDAS  shared  drives  are  temporary  while  the  program 
migrates  from  VDD  to  LCIS.  Risk  owners  are  responsible  for 
updating  risk  mitigation  activities  and  assessments  just  as 
in  PDRR.  DRPM  AAA  and  GDAS  personnel  may  access  and  review 
all  risk  forms  contained  on  the  shared  drives.  Figure  29 
depicts  an  example  AAAV  SDD  risk  assessment  form. 

While  the  AAAV  program' s  PDRR  risk  management 
process  resulted  in  the  successful  deployment  of  three 
prototypes  used  in  testing  and  production  refinement,  the 
PMO  now  benefits  from  lessons  learned  during  PDRR. 
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Figure  29.  AAAV  SDD  Risk  Assessment  Form,  from 

(AAAV  Risk  Mitigation  Planning  Guidance  for  the 
Risk  Owner,  November  2002)  . 

Those  lessons  learned  are  already  helping  shape 
and  improve  the  risk  management  process  during  SDD.  The 
next  section  discusses  several  lessons  learned  from  the 
PDRR  risk  management  process  and  introduces  what  practices 
the  program  has  adopted  as  a  result  of  the  lessons  learned. 
b.  AAAV  SDD  Risk  Management  Process 
The  goal  of  an  effective  risk  management  process 
during  SDD  is  to  deliver: 

•  Fewer  unexpected  costs 

•  Better  cost  control 

•  Improved  adherence  to  program  schedule 

•  Enhanced  vehicle  performance  (Risk  Coalition  Team 

Brief,  June  2002) 
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The  current,  SDD  contract  requires  GDAS  to 
develop  for  the  Government's  approval  a  formal  risk 
management  process.  The  process  used  during  PDRR  focused 
primarily  on  technical  and  system-level  risks.  When 
consistently  implemented,  the  PDRR  risk  management  process 
effectively  helped  the  AAAV  Team  to  manage  technical  risks. 
Similar  to  the  PDRR  phase,  the  GDAS  SDD  risk  management 
plan  will  focus  mainly  on  technical  and  system-level  risks: 
areas  that  are  within  the  scope  of  their  contractual 
effort . 

However,  the  DRPM  AAA  recognized  that  Government - 
specific  risks  exist  in  SDD  that  are  not  adequately 
addressed  or  managed  through  a  risk  management  process 
developed  and  implemented  by  the  Contractor  like  that  used 
in  PDRR.  The  types  of  risks  in  SDD  are  not  the  same  types 
of  risks  encountered  during  PDRR.  There  are  more 
programmatic  risks,  which  are  not  within  the  scope  of  GDAS' 
effort.  Accordingly,  the  DRPM  as  well  as  the  GDAS  risk 
management  processes  needed  to  adapt  to  the  program 
conditions . 

As  a  result,  the  Government  is  taking  a  more 
active  leadership  role  during  SDD  to  manage  Government - 
specific  risks  separately  from  system-level,  technical  or 
developmental  Contractor  risks.  DRPM  AAA  is  modifying 
their  risk  management  plan  to  enhance  the  focus  on 
Government-specific  risks  such  as  funding,  testing  and  life 
cycle  considerations:  those  risks  outside  GDAS'  "sphere  of 
influence."  (Risk  Coalition  Team  Brief,  June  2002) 
Meanwhile,  GDAS  is  contractually  obligated  to  develop  and 
implement  a  risk  management  process  for  Government  approval 
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and  oversight  through  production.  The  portion  of  the 
Government's  plan  for  technical  risks  will  leverage  off  of 
GDAS'  plan  since  both  the  Government  and  GDAS  will  actively 
manage  the  technical  risks  throughout  the  SDD  phase.  GDAS' 
risk  management  is,  again,  embedded  in  contract  award  fee 
criteria  during  SDD. 

A  lesson  learned  from  PDRR  is  that  risk 
management  is  a  significant,  continuous  effort  that 
requires  "motivated  personnel  coordinating  risk."  (Kepner, 
EPMC  Brief)  Additionally,  the  frequency  of  risk  management 
meetings  must  coincide  with  program  activities  and  the  need 
for  increased  attention.  For  example,  during  the  whole 
systems  and  subsystems  trade  processes,  it  is  critical  that 
risks  be  understood  and  managed  as  the  system  parameters 
and  component  capabilities  are  established.  (Kepner,  2003) 
The  risk  management  process,  once  approved  and  implemented, 
must  be  sustainable  during  all  periods  of  SDD. 

The  Independent  Risk  Assessment  Team  (IRAT) 
conducted  prior  to  the  MSII  decision  indicated  that  formal 
risk  management  practices  waned  during  periods  just  before 
significant  program  events  (i.e.,  prototype  roll  out,  major 
test  events,  etc.)  during  PDRR.  Programmatic  focus  on 
achieving  key  milestones  temporarily  impacted  "systems 
engineering  processes"  (Kepner,  EPMC  Brief)  Systems 
engineering  processes  include  risk  management,  requirements 
traceability,  synthesis,  etc.  The  IRAT  recommended  that 
process  sustainability  be  a  key  characteristic  of  the  SDD 
risk  management  process.  Accordingly,  the  DRP  M  AAA 

directed  that  a  program  directorate  assume  risk  management 
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process  coordination  consistent  with  the  activities 
normally  conducted  during  SDD . 

The  AAAV  Team  formalized  a  functional,  program 
office  directorate  called  Program  Planning  and  Integration 
(PP&I)  .  PP&I  is  responsible  for  the  AAAV  risk  management 
plan  and  its  implementation  during  SDD.  The  Program 
Integration  Division  Head  is  part  of  PP&I  and  is  the 
program  Risk  Management  Coordinator  (RMC) .  Currently,  both 
Government  and  GDAS  members  are  refining  the  formal  SDD 
risk  management  plan.  The  Government  side  of  PP&I  also 
focuses  on  Government-specific  risks  and  advises  the  AAAV 
Program  Management  Team  (PMT)  .  The  PMT  consists  of  the  PM, 
Deputy  PM  and  Government  Department  Heads  (i.e..  Test  and 
Evaluation  Head,  Manufacturing  Head,  Lead  Engineer,  PP&I, 
etc . )  . 

The  RMC  chairs  a  bi-weekly  meeting  to  address 
programmatic  issues  at  the  Department  Head  level  called  the 
Program  Management  Team  (PMT)  II.  One  of  the  focal  points 
of  the  PMT  II  meeting  is  cross -functional  risk 
communication  and  courses  of  action  development.  The 
purpose  is  to  enhance  communication  of  risks  at  the  action 
officer  level  in  the  program  and  support  decision-making  at 
the  next  highest  level,  the  PMT  level.  The  PMT  II  advises 
the  PMT  on  risk  mitigation  alternatives  and  resource  impact 
for  decision-making  support. 

The  PMT  II  is  well  suited  to  analyze  system-wide 
metrics  to  assess  the  success  of  mitigation  activities. 
Additionally,  the  meeting  is  used  as  a  venue  to  discuss 
award  fee  determinations  and  recommendations  for  upcoming 
contract  performance  assessment  reviews. 
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The  need  for  the  PMT  II  is  essential  during  the 
SDD  stage  when  the  program  transitions  from  development  to 
production  activities.  The  increase  in  the  number  of 
program  personnel  makes  communication  even  more  critical. 
Additionally,  the  PMT  II  can  perform  many  of  the  same 
functions  that  the  Risk  Resolution  Board  (RRB)  did  during 
PDRR  without  some  of  the  drawback  encountered  with  the  RRB. 

The  RRB  was  an  effective  forum  to  analyze  risk 
areas  and  serve  as  a  decision-making  board  at  the  highest 
program  office  level  during  PDRR.  The  RRB  process  ensured 
dissemination  of  risks  throughout  the  program  from  D  Level 
IPTs  to  the  PMT.  However,  amid  the  RRB' s  successes,  two 
challenges  eventually  emerged  that  caused  the  AAAV  program 
to  adopt  a  different  strategy  during  SDD. 

First,  because  the  RRB  relied  on  the  senior 
program  leadership,  its  frequency  of  meeting  became 
difficult  during  especially  busy  time  periods  of  PDRR.  The 
PDRR  risk  management  process  depended  on  the  RRB  to  provide 
guidance  and  decision-making  to  function  efficiently  and 
consistently.  The  RRB,  though  a  good  idea  and  effective 
when  executed,  lacked  the  sustainability  characteristic 
needed  in  SDD. 

Secondly,  some  personnel  from  DRPM  AAA  and  GDAS 
believed  the  RRB  became  a  forum  that  did  not  always  embrace 
risks  as  opportunities  or  foster  an  environment  of  open, 
non-attribution  discussion.  One  GDAS  employee  said  that 
the  RRB  became  a  "moot  court . "  The  result  was  a 
disincentive  for  IPTs  or  stakeholders  to  identify  and 
introduce  risks  into  the  risk  management  process.  The  DRPM 
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AAA  has  responded  by  establishing  the  PMT  II  meetings, 
which  serve  as  the  Risk  Coalition  Team  (RCT) . 

The  RCT  is  "a  team  of  Government  personnel 
chartered  as  owners  of  the  risk  management  process.  Its 
primary  purpose  is  to  facilitate  the  creation  and 
continuous  operation  of  the  risk  management  process." 
(Draft  DRPM  AAA  SDD  Risk  Management  Plan,  November  2002) 
The  SDD  RMC  (Program  Integration  Branch  Head)  chairs  the 
RCT.  The  RCT  includes  members  from  the  following  DRPM  AAA 
directorates : 

•  PP&I 

•  Systems  Engineering 

•  Logistics 

•  Testing 

•  Cost 

•  Engineering  representatives 

•  DCMA  representatives  (Draft  DRPM  AAA  SDD  Risk 

Management  Plan,  November  2002) 

The  biggest  difference  between  the  RRB  used  in 
PDRR  and  the  RCT  used  in  SDD  is  the  organization' s 

membership.  The  RRB  consisted  mainly  of  Division  Directors 
whereas  the  RCT  is  comprised  of  the  next  lower  level,  the 
Department  Heads  or  PMT  II  members.  The  benefit  in  the 

revised  structure  is  that  the  RCT  performs  the  majority  of 
detailed  risk  analysis  so  they  can  then  provide 

recommendations  and  courses  of  action  to  the  PMT.  The  RCT 
can  filter  and  solve  many  problems  before  coming  to  the 
attention  of  the  PMT  allowing  the  PMT  to  focus  on  higher 
level.  Government -specif ic  risk  areas.  The  RCT  meets  bi¬ 
weekly  as  part  of  the  PMT  II  meetings. 
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Another  risk  management  technique  the  DRPM  AAA 
used  during  PDRR  was  inviting  an  Independent  Risk 
Assessment  Team  (IRAT)  .  The  purpose  of  the  IRAT  in  PDRR 
was  to  evaluate  the  effectiveness  of  the  program' s  risk 
management  process  and  provide  recommendations  for  SDD . 

c.  Periodic  Risk  Assessments 

During  PDRR,  the  AAAV  Team  conducted  periodic 
internal  and  external  risk  assessments  of  the  program' s 
risk  management  process.  Prior  to  Milestone  II,  the  AAAV 
Team  requested  an  independent  risk  assessment  to  evaluate 
the  status  of  the  program' s  technical  risks  and  the  risk 
management  process  prior  to  entry  into  the  System 
Development  and  Demonstration  (SDD)  Phase. 

EG&G,  DRPM  AAA' s  risk  management  support 
contractor,  nominated  a  "three-member  team  of  experienced 
engineers"  to  conduct  the  assessment.  The  Independent  Risk 
Assessment  Team  (IRAT)  was  made  up  of  EG&G  Technical 
Services  representatives  and  chaired  by  a  representative 
from  the  Illinois  Institute  of  Technology  Research 
Institute  who  were  also  co-authors  of  the  NAVSO  Guide  "Top 
Eleven  Ways  to  Manage  Technical  Risks."  (IRAT  Report, 
September  2000) 

The  DRPM  AAA  tasked  the  IRAT  with  conducting  an 
impartial  assessment  of  the  AAAV  risk  management  program. 
The  AAAV  DRPM  planned  to  integrate  the  IRAT' s  findings  into 
part  of  the  Milestone  II  Defense  Acquisition  Board  (DAB) 
documentation.  (AAAV  Independent  Risk  Assessment  Brief, 
March  2000) 

The  risk  areas  of  interest  were  product  and 
technical  process  risks.  (IRAT  Report,  2000)  The  AAAV  Team 
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intended  to  use  the  IRAT  findings  to  prepare  for  the 
upcoming  Milestone  II  decision,  identify  technical  and 
process  risks  associated  with  entering  SDD  and  evaluate 
risk-handling  procedures  during  PDRR.  The  DRPM  provided 
IPT  briefings  and  disclosed  extensive  technical  and  program 
documentation  to  facilitate  the  assessment. 

The  IRAT' s  methodology  was  as  follows: 

•  Review  related  program  documents,  such  as  AAAV 

Risk  Management  Plan,  SEMP,  Management  Plan,  ORD, 

TEMP  and  both  the  Program  Definition  and  Risk 

Reduction  (PDRR)  and  EMD  SOWs 

•  DRPM  AAA  IPT  leads  briefed  the  IRAT  regarding 
implementation  of  risk  management,  issues/risks, 
and  status 

•  Follow-up  with  interviews,  discussions  and  data 

gathering  in  selected  areas 

•  Prepare  the  IRAT  report  and  out  brief  results  to 
the  DRPM  AAA  (IRAT,  September  2000) 

The  objective  of  the  assessment  was  to  provide  an 

objective  overview  of  the  program's  risk  management  process 

and  identify  risk  areas  for  entry  into  SDD. 

The  DRPM  AAA  found  the  IRAT  to  be  extremely 
useful.  The  IRAT' s  objectivity  on  the  risk  management 
process  provided  invaluable  feedback  that  allowed  the  AAAV 
program  to  shape  its  future  SDD  processes.  The  DRPM  AAA 
intends  to  conduct  periodic  IRATs  during  SDD  and  in 
preparation  for  the  TRIP  decision. 

The  risk  management  process  discussed  in  the 
preceding  section  analyzed  and  introduced  several  of  AAAV' s 
risk  management  techniques  used  during  PDRR  and  SDD.  The 
next  analyzes  one  aspect  of  the  AAAV  program' s  use  of  the 
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contracting  process  to  reduce  programmatic  risks  and 
transfer  risk  from  the  Government  to  the  contractor. 

3 .  AAAV  Risk  Management  Through  the  Contracting 
Process 

Following  the  Milestone  I  decision,  the  Government 
awarded  GDAS  a  Cost  Plus  Award  Fee  (CPAF)  type  contract  for 
PDRR.  Part  of  the  second  period  award  fee  criteria  was 
tied  to  GDAS'  risk  management  process.  Using  contract 
award  fees  to  provide  contractors  with  incentives  to 
proactively  manage  risk  is  an  effective  risk  mitigation 
activity . 

GDAS  uses  an  award  fee  sharing  system  among  its 
employees.  Each  GDAS  employee  earns  a  portion  of  the 
contract  award  fee.  A  graduated  scale  determines  the 
various  amounts.  Employees  earn  percentages  of  award  fees 
according  to  their  position  within  GDAS.  Award  fee 
sharing,  or  profit  sharing,  in  organizations  motivates  good 
behavior  and  provides  incentive  for  all  contractor 
employees  to  perform. 

As  a  result  of  the  award  fee  assessment,  GDAS  placed 
higher  priority  on  risk  management,  from  both  a  capability 
as  well  as  training  aspects.  It  was  at  this  time  that  GDAS 
developed  the  VDD  risk  management  applications  and 
formalized  the  process  used  during  PDRR. 

The  lesson  learned  was  that  contractually  obligating 
and  providing  incentives  for  the  Prime  contractor  to 
spearhead  risk  management  efforts  transferred  some 
accountability  for  risk  from  the  Government  to  industry. 
The  Government  was  then  prepared  to  evaluate  the  Prime 
Contractor's  performance  and  reward  them  accordingly 
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through  contract  period  award  fees.  The  current  SDD 
contract  type  is  also  a  CPAF  with  award  fee  criteria  tied 
to  GDAS'  risk  management  process.  This  also  allows  the 
Government  to  supervise  GDAS'  technical  and  system-level 
risk  management  while  concurrently  implementing  a 
Government-specific  risk  plan. 

GDAS'  award  fee  sharing  system  sustains  employee  buy- 
in  by  financially  rewarding  its  people  for  good 
performance.  The  DRPM  AAA's  contracting  strategy  is  a  good 
example  of  risk  transference  in  DoD  systems  acquisitions. 
Once  implemented,  the  AAAV  risk  management  process  relied 
heavily  on  Integrated  Product  Team  (IPT)  interaction  and 
the  Integrated  Product  and  Process  Development  (IPPD) 
process.  The  next  section  analyzes  the  AAAV  program's 
unique  co-location  and  IPPD  organization. 

4.  AAAV  PDRR  Co-Location  and  IPPD  Process 

The  AAAV  program' s  co-location  at  the  Woodbridge,  VA 
facility  fosters  continual  communication  and  interaction 
between  Government  and  GDAS  personnel.  This  author 

interviewed  both  DRPM  AAA  and  GDAS  individuals  concerning 
co-location.  Both  groups  were  in  agreement  that  co- 
location  allows  for  a  great  degree  of  real-time 
communication,  which  can  reduce  design  risks,  especially  in 
the  PDRR  phase.  Synthesis  is  a  vital  step  in  the  systems 
engineering  process  and  communication  is  imperative  to 
effective  synthesis,  as  decisions  are  being  made  with  the 
full  understanding  of  the  Government  through  its 
counterparts  on  the  IPTs.  In  addition  to  the  communication 
benefits,  both  DRPM  and  GDAS  personnel  agreed  that  co- 
location  helps  each  group  understand  the  other' s  culture 
and  therefore  develop  more  effective  working  relationships. 
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In  addition  to  the  office  space  co -location,  the  vehicle 
assembly  and  production  area  located  at  the  joint  facil  ity 
was  of  great  value. 

Co-location  at  the  point  of  vehicle  assembly  allowed 
nearly  seamless  system  integration  activities.  This 
facility  reduced  the  risk  of  engineering  and  schedule 
problems  by  allowing  GDAS  and  DRPM  decision-makers  to  apply 
timely  resources  to  friction  points  and  fully  understand 
the  risks  being  mitigated. 

During  a  visit  to  the  Worth  Avenue  Technology  Annex  in 
Virginia,  this  author  saw  active  duty  Marines, 
representative  of  the  intended  end-users,  providing  input 
during  design  and  assembly  of  SDD  prototypes.  This  type  of 
interaction  greatly  reduces  the  risk  of  systems  not 
adequately  accounting  for  logistics  and  supportability 
considerations  during  the  design  stages.  Had  GDAS  and  DRPM 
AAA  not  been  co-located,  it  is  doubtful  that  user 
representatives  would  have  as  much  opportunity  to  provide 
input  and  feedback  in  the  design  stages.  The  alternative 
can  result  in  time-consuming  and  costly  system  changes 
prior  to  fielding  or  once  a  system  is  fielded. 
Additionally,  funding  usually  necessary  for  travel  to  bring 
IPTs  or  program  leaders  together  is  avoided  by  co -locating. 

Co-location  is  synonymous  with  the  principles  of 
Integrated  Product  and  Process  Development  (IPPD) . 
Continual  cross -functional  area  communication  is  a  large 
part  of  the  IPT  process.  Co-location  allows  the  AAAV 
program  to  operate  in  an  IPPD  environment . 
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a.  Integrated  Product  and  Process  Development 
(IPPD)  and  the  AAAV  Program 

The  AAAV  Team' s  co-location  facilitates  the  IPPD 
process  and  allows  IPTs  to  work  and  interact  on  a  frequent 
basis  during  SDD .  The  AAAV  co-location  as  an  enabler  of 
the  IPPD  process  "provides  the  foundation  and  communication 
conduits  for  the  IPTs  to  maximize  the  effectiveness  of 
every  member  of  the  organization."  (Pollution  Prevention: 
AAAV,  October  2002)  The  Government  and  GDAS  co-location 
facilitates  IPT  integration  and  interaction  between  the 
Government  and  company  engineers,  logisticians,  product 
managers,  and  other  functions.  The  program's  emphasis  on 
cross-functional  coordination  and  efforts  has  and  will 
result  in  significant  schedule  and  cost  risk  mitigation  and 
avoidance.  In  particular,  the  AAAV' s  dedication  to 

reducing  and,  in  some  cases,  eliminating  environmental 
safety  hazards  (ESH)  in  the  system' s  production  will  reap 
noteworthy  Total  Ownership  Cost  (TOC)  and  Lifecycle  Cost 
(LCC)  avoidances.  Each  AAAV  IPT  has  an  ESH  representative. 
The  following  section  is  a  case  study  of  the  benefits  of 
IPT  co-location  and  IPPD  interaction. 

b.  Environmental  Safety  and  Health  (ESH)  and 
the  IPPD  Process 

Environmental  Safety  and  Health  (ESH) 
considerations  may  constitute  significant  programmatic 
risks.  Such  risks  include  safety  of  use,  environmental 
impact  of  system  use  and  exorbitant  system  disposal  costs. 
In  response  to  environmental  risk  considerations,  the  AAAV 
DRPM  established  an  ESH  Working  Group  (ESH  WG) .  The  ESH  WG 
was  tasked  to  "identify,  evaluate,  track  and  assist  with 
mitigation  of  ESH  hazards."  (Pollution  Prevention:  AAAV, 
October  2002)  The  ESH  WG' s  membership  included  both 
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Government  and  GDAS  personnel  who  possess  experience  with 
ESH  issues  and  environmental  considerations.  AAAV  IPTs  all 
have  ESH  representatives.  The  representatives  ensure  that 
ESH  considerations  and  impact  are  part  of  all  development 
and  production  decisions  in  the  IPPD  process. 

The  ESH  WG  created  the  "first  ever  Risk  Reduction 
Process,  embedded  in  the  VDD,  to  identify,  track,  and 
eliminate  ESH  hazards."  (Pollution  Prevention:  AAAV, 
October  2002)  The  ESH  WG  identified  and  assigned  "over 
five  hundred  ESH  risk  hazards"  and  "developed  the  ESH 
Database  that  provides  the  communication  and  tracking  link 
for  these  hazards,  the  identification  of  the  lead  IPT  for 
mitigation  action,  and  tracking  of  the  risk  hazard 
resolution/acceptance."  (Pollution  Prevention:  AAAV, 
October  2002) 

The  DRPM  AAA  developed  an  ESH  Awareness  Session 
for  all  members  in  AAAV  IPTs.  (Pollution  Prevention:  AAAV, 
October  2002)  The  purpose  of  the  ESH  awareness  session  was 
to  train  and  educate  AAAV  Team  members  on  the  potential 
risks  associated  with  ESH. 

The  DRPM  AAA  drafted  the  system' s  performance 
specifications  to  include  a  ban  on  all  Class  I  and  II  Ozone 
Depleting  Substances  (ODS)  in  the  design  and  manufacture  of 
the  AAAV.  Additionally,  the  Government  has  contractually 
obligated  both  Prime  and  Subcontractors  to  eliminate  the 
use  of  cadmium,  lead,  chromium  and  other  environmentally 
hazardous  materials  in  the  production  of  the  AAAV.  The 
deletion  of  these  environmentally  harmful  substances  will 
reduce  the  risks  of  negative  environmental  impacts  and  high 
disposal  costs. 
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Over  the  life  of  the  AAAV  program,  the  AAAV 
anticipates  cost  avoidance  of  $379.9  million  in  production 
and  $238.9  million  in  Operations  and  Support  (O&S)  costs. 
(Pollution  Prevention:  AAAV,  October  2002)  The  AAAV  ESH 
initiatives  have  played  a  significant  role  in  the  projected 
cost  avoidances  through  the  application  of  the  ESH  Working 
Group  (ESH-WG)  and  the  Virtual  Design  Database. 

The  U.S.  Army's  Center  for  Health  Promotion  and 
Preventive  Medicine  (CHPPM)  and  the  U.S.  Navy's  Explosive 
Safety  Review  Board  (WSESRB)  stated  that  the  AAAV' s  "ESH-WG 
Risk  Reduction  Process  is  the  best  program  of  its  type  that 
they  have  encountered  in  DoD . "  (Pollution  Prevention:  AAAV, 
October  2002)  The  ESH-WG  IPT  representation  ensures  that 
ESH  considerations  are  input  throughout  every  design  and 
manufacturing  decision  made  for  the  AAAV.  Through  the 
system  decomposition  and  iterative  design  process,  ESH 
input  in  the  IPT  process  enhances  the  likelihood  that  the 
system  is  built  correctly  the  first  time.  This  eliminates 
costly  design  or  manufacturing  process  changes  after  the 
system  is  fielded  or  prior  to  demilitarization.  The  ESH-WG 
and  ESH  IPT  representation  are  examples  of  the  benefits  of 
the  IPPD  process  in  weapon  system  developments.  ESH  risks 
were  incorporated  into  the  program' s  risk  management 
process  during  PDRR  in  the  same  manner  as  all  other  risks 
managed  by  IPTs. 

The  relational  VDD,  discussed  in  preceding 
sections,  allowed  the  entire  AAAV  IPT  structure  to  have 
visibility  on  ESH  risk  identification,  tracking,  resolution 
and  documentation  issues.  The  VDD  has  resulted  in 
effective  horizontal,  vertical  and  cross  communications 
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concerning  ESH  risk  handling.  Because  risks  in  VDD 
corresponded  to  WBS  elements,  the  cause  and  effect 
relationships  of  ESH  risks  were  easy  to  identify  and 
assess . 

The  AAAV  program  has  taken  proactive  measures  to 
reduce  the  system' s  environmental  impact .  These  measures 
will  lead  to  cost  avoidance  over  the  life  of  the  program  as 
well  as  preserving  the  environment.  The  continual  ESH 
assessment  during  the  IPT  process  may  have  been  less 
effective  or  ineffective  had  the  AAAV  program  not  been  co¬ 
located  during  PDRR.  The  program's  co-location  and 
commitment  to  IPPD  enabled  ESH  considerations  to  be  part  of 
design  trade  offs  and  CAIV  analysis . 

The  AAAV  performance  specifications  negate  the 
use  of  several  environmentally  hazardous  materials  in  the 
system's  production.  Offerors  have  had  to  develop  and 
incorporate  new,  environmentally  safe  materials  for 
integration  into  the  AAAV.  For  example,  cadmium  was 
eliminated  because  of  the  presence  of  cyanide  and  other 
hazardous  materials  (HAZMAT)  in  the  plating  process. 
(Pollution  Prevention:  AAAV,  October  2002) 

Furthermore,  the  AAAV  is  testing  a  developmental, 
water-based  Chemical  Agent  Resistant  Coating  (CARC)  paint 
that  will  reduce  Hazardous  Air  Pollutants  (HAP)  during 
manufacturing  and  repair  of  the  system.  The  use  of  a 
water-based  CARC  paint  is  expected  to  save  $2 . 8  million 
over  the  life  of  the  program.  (Pollution  Prevention:  AAAV, 
October  2002)  The  new  CARC  paint  may  represent  enormous 
savings  throughout  DoD  for  systems  that  require  CARC.  Such 
developmental  innovations  may  reduce  future,  developmental 
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system  costs  and  reduce  the  risk  of  acquisition  funds  being 
diverted  from  procurement  to  support  unanticipated  O&S 
costs . 

Working  with  the  U.S.  Environmental  Protection 
Agency  (USEPA)  ,  the  AAAV  program  has  also  eliminated  the 
use  of  all  Class  I  and  II  Ozone  Depleting  Substances  (ODS) 
and  the  majority  of  the  USEPA' s  top  seventeen  hazardous 
materials .  Savings  as  a  result  of  the  elimination  of  these 
hazards  is  expected  to  be  in  the  tens  of  millions  of 
dollars  with  even  greater  potential  with  DoD-wide  adoption. 

Reduction  of  HAZMAT  initiatives  during  the  AAAV' s 
design  and  manufacture  directly  impacts  Total  Ownership 
Costs  (TOC)  .  The  AAAV' s  avoidance  of  ODS  and  other  HAZMAT 
will  allow  the  system  to  enter  its  disposal  phase  without 
significant  commitment  of  additional  resources  to  make  the 
disposal  environmentally  friendly.  The  AAAV' s  insistence 
that  the  Prime  and  Subcontractors  use  environmentally 
improved  materials  will  result  in  savings  in  future  systems 
LCC  and  Research,  Development,  Test  and  Evaluation  (RDT&E) 
activities . 

ESH  risk  mitigation  is  just  one  example  of  the 
AAAV  IPPD  process.  Although  other  DoD  programs  that  are 
not  co-located  effectively  operate  in  an  IPPD  environment, 
the  AAAV' s  co-location  encourages  continual,  cross¬ 
functional  area  communication  and  constant  interaction 
between  GDAS  and  the  DRPM  AAA.  In  order  to  determine  the 
effectiveness  and  supportability  of  ESH  initiatives,  the 
AAAV  team  needs  to  thoroughly  test  the  system.  The  next 
section  analyzes  the  AAAV  PDRR  test  and  evaluation  plan. 
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5 .  AAAV  PDRR  Test  and  Evaluation 

The  aggressive  developmental  testing  (DT)  and  early 
operational  assessment  (EOA)  performed  during  PDRR  will 
benefit  the  AAAV  program  during  SDD  and  subsequent 
production  and  fielding.  The  results  of  the  DT  events  and 
the  EOA  in  PDRR  allowed  the  AAAV  team  to  identify  what 
areas  of  the  system  required  further  developmental 
attention  and  testing.  The  PDRR  test  results  helped 
determine  the  SDD  test  plan.  On  using  test  as  a  risk 
management  tool,  the  DAU  writes: 

Fixes  instituted  during  early  work  efforts 
(Systems  Integration)  in  the  System  Development 
and  Demonstration  (SDD)  Phase  cost  significantly 
less  than  those  required  in  later  System 
Demonstration  after  the  critical  design  review 
when  most  design  decisions  have  been  made." (Test 
and  Evaluation  Management  Guide,  November  2001) 

Early  testing  can  reduce  the  risk  of  run-away  system 
delivery  costs  and  expensive  design  changes  late  in  the 
acquisition  process.  By  conducting  an  EOA  during  a 

Combined  Arms  Exercise  (CAX)  in  29  Palms,  California,  the 
AAAV  program  combined  developmental  and  operational  test 
(OT)  activities.  DT  under  operational  conditions,  similar 
to  those  specified  in  the  Operational  Requirements  Document 
(ORD) ,  can  reduce  time  and  costs  through  concurrent 
testing.  When  conducted  too  late  in  a  program's  schedule, 
combined  DT  and  OT  events  can  result  in  OT  failures  and  can 
impact  milestone  decisions.  Conducted  early,  the  program 
can  identify  risk  areas  and  develop  mitigation  and  test 
plans  to  fix  and  validate  deficiencies.  The  DoD  5000. 2 -R 
encouraged  combined  DT  and  OT  testing: 
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A  combined  developmental  test  and  evaluation 
(DT&E)  and  operational  test  and  evaluation  (OT&E) 
approach  should  be  considered  when  there  are  time 
and  cost  savings .  The  combined  approach  must  not 
compromise  either  DT  or  OT  objectives.  (DoD 
5000. 2-R,  April  2002) 

Data  obtained  from  combined  DT  and  OT  events  can  be 
collected  and  potentially  used  in  lieu  of  follow-on  test 
events.  Coordination  with  service  test  agencies  and  the 
Director  of  Operational  Test  and  Evaluation  (DOT&E)  is 
important  for  obtaining  and  using  test  data. 

Another  lesson  learned  from  the  AAAV  gunnery  EOA  was 
that  crew  training  and  experience  on  the  weapon  system  is 
critical  to  successful  test  execution.  The  PMO  is  applying 
this  lesson  learned  to  the  test  crew-training  plan  during 
SDD . 

In  addition,  the  program  office  conducted  extensive 
testing  on  the  MK46  weapons  station  to  include  a  gunnery 
EOA.  The  MK46  lethality  tests  met  or  exceeded  all  ORD 
requirements  during  PDRR.  In  the  course  of  testing,  the 
AAAV  program  identified  areas  that  will  require  additional 
study  and  follow-on  testing  in  SDD  such  as  ventilation  and 
gunner  training.  This  knowledge  helped  the  DRPM  AAA 
develop  the  SDD  test  plan  and  concentrate  on  specific 
system  components  prior  to  additional  OT . 

Other  than  the  expected  system  performance -oriented 
test  events,  the  AAAV  program  conducted  extensive  life 
cycle  support  testing  during  PDRR. 

Life  cycle  support  testing  included  logistics 
demonstrations,  maintainability  demonstrations,  mean  time 
to  repair  (MTTR)  demonstrations  and  human  factors 
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engineering  testing.  The  significance  of  the  AAAV' s  PDRR 
test  plan  was  its  emphasis  on  life  cycle  support.  Life 

cycle  support  considerations  will  drive  the  AAAV' s  life 
cycle  costs  (LCC) .  Some  estimates  report  that  nearly  75 
80%  of  a  system's  overall  cost  is  assumed  during  Operations 
and  Support  (O&S) . 

Test  events  designed  to  validate  ORD-specified  MTTR  or 
operational  availability  parameters  can  reduce  LCC  and  poor 
operational  availability.  This  can  be  achieved  by 

including  user  juries  during  logistics  demonstrations  and 
MTTR  demonstrations.  User  juries  can  identify  system  re¬ 
design  recommendations  to  make  the  system  more 
maintainable.  The  system's  supportability  will  drive  its 
operational  availability.  Decreased  repair  requirements 
and  cycle  times  will  reduce  maintainability  costs  and  time. 
The  result  will  be  a  supportable  and  available  system  for 
the  intended  user. 

Logistics  can  be  a  primary  system  cost  driver  during 
O&S.  Thoroughly  testing  a  system  during  PDRR  and  SDD  for 
supportability,  reliability  and  maintainability  can  greatly 
reduced  O&S  costs  and  increases  operational  availability  by 
identifying  supportability  risks  early  on  in  the  program' s 
schedule.  Reducing  the  risk  of  unanticipated  O&S  costs  can 
protect  resources  intended  for  pre-planned  program 
improvements  (P3I)  and  new  developmental  systems. 

The  SDD  test  plan  will  focus  on  verifying  design 
improvements  identified  in  PDRR  by  testing  an  expected  nine 
SDD  prototypes.  (Developmental  Testing  Brief,  November 
2002)  The  emphasis  will  be  on  addressing  design  changes 
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identified  in  PDRR  and  conducting  OT  activities  in  order  to 
meet  Low  Rate  Initial  Production  (LRIP)  entrance  criteria. 

E.  SUMMARY 

This  chapter  analyzed  the  data  presented  in  Chapter  IV 
of  this  thesis.  This  chapter  discussed  the  author's 
purpose  of  research  and  research  methodology.  The  primary 
purposes  were  to  analyze  and  discuss  the  AAAV  PDRR  risk 
management  strategy  and  connect  the  lessons  learned  in  PDRR 
to  the  SDD  risk  management  techniques. 

This  chapter  discussed  AAAV  PDRR  information 
technology  tools:  VDD,  VINTEGRA  and  LCIS.  The  lesson 
learned  from  PDRR  was  that  information  technology  tools 
could  be  force  multipliers  in  managing  risk.  VDD  served  as 
a  central  repository,  or  risk  database.  VDD  risk 
management  applications  facilitated  the  formal  risk 
management  process.  VDD's  automatic  notification  system 
enabled  communication  across  program  functional  lines. 
Shortfalls  in  VDD  have  led  to  the  development  of  LCIS 
during  SDD.  LCIS  aims  to  create  a  risk  management  tracking 
application  that  emphasizes  trend  analyses,  reporting  and 
program-wide  risk  management. 

VINTEGRA  continues  to  reduce  system  production  risks 
by  identifying  integration  and  assembly  and  production 
refinement  requirements.  Additionally,  VINTEGRA  will 
reduce  cost  risks  in  the  development  of  IETMs .  The  program 
office  expects  IETMs  to  reduce  maintenance  delay  times  thus 
improving  operational  availability  and  logistics  strains 
during  O&S. 

The  AAAV  PDRR  risk  management  process  was  effective. 
However,  the  program  office  learned  valuable  lessons  from 
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its  process  and  is  applying  lessons  learned  to  the  process 
used  in  SDD.  Namely,  the  DRPM  AAA  and  GDAS  instituted  the 
PP&I  directorate.  One  of  the  functions  within  PP&I  is  the 
overall  risk  management  process.  The  program  implemented 
the  PMT  II  bi-weekly  meetings.  Risk  management  is  a 
statutory  PMT  II  agenda  item.  Through  the  PMT  II,  the  RCT 
will  manage  program  risks  and  develop  courses  of  action  to 
present  to  the  program  leadership  for  decision  support. 

During  PDRR,  the  Government  incentivized  risk 
management  through  the  use  of  an  award  fee.  The  CPAF 
contract  type  provided  financial  incentives  for  GDAS  to 
execute  sound  risk  management  practices.  The  Government 
awarded  GDAS  a  CPAF  type  contract  for  SDD  with  risk 
management  tied  to  award  fee  criteria,  as  well.  Including 
financial  incentives  in  contract  award  fee  criteria  is  an 
effective  technique  to  transfer  risk  from  the  Government  to 
a  Prime  contractor.  Both  entities  gain  from  effective  risk 
handling  as  a  result. 

The  DRPM  AAA  has  co-located  with  its  Prime  contractor, 
GDAS,  in  Northern,  Virginia.  The  AAAV  program  co -location 
has  facilitated  continual  communication  and  interaction 
between  Government  and  Industry  personnel.  The  close 
working  relationship  is  consistent  with  IPPD  principles. 
The  AAAV  program' s  IPPD  process  has  benefited  from  the 
communication  advantages  provided  by  co -location. 

Finally,  the  AAAV  PDRR  test  plan  included  OT 
activities  combined  with  DT .  The  aggressive  test  plan 
allowed  the  program  to  identify  areas  that  will  require 
greater  attention  during  SDD.  The  results  of  the  PDRR  DT, 
OT  and  EOA  have  helped  shape  the  SDD  test  plan.  The  SDD 
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test  strategy  is  to  fix  design  deficiencies  discovered 
during  PDRR  and  prepare  the  system  to  meet  LRIP  entrance 
criteria . 

The  next  chapter  concludes  this  thesis.  The  purpose 
of  the  conclusion  is  to  discuss  how  the  AAAV  SDD  risk 
management  strategy  reflects  the  lessons  learned  from  the 
PDRR  risk  management  approach.  Additionally,  the  chapter 
discusses  which  elements  of  risk  management  practices  the 
AAAV  program  is  benefiting  from  most  and  provides 
recommendations  for  managing  risk  in  developmental  weapon 
systems . 
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VI .  CONCLUSION 


A.  INTRODUCTION 

This  chapter  provides  conclusions  and  recommendations 
drawn  from  the  analysis  of  the  Advanced  Amphibious  Assault 
Vehicle  (AAAV)  Program  Definition  and  Risk  Reduction  (PDRR) 
and  System  Development  and  Demonstration  (SDD)  risk 
management  strategies.  The  benefit  of  this  research  is  to 
illustrate  risk  management  techniques  used  in  Department  of 
Defense  (DoD)  weapon  system  procurement  and  development 
through  a  study  of  the  AAAV' s  transition  from  PDRR  to  SDD. 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

This  section  discusses  conclusions  regarding  risk 
management  procedures  used  by  the  AAAV  program  during  SDD. 
Based  on  the  conclusions,  this  chapter  offers 
recommendations  for  managing  risk  in  DoD  developmental 
programs . 

1 .  AAAV  Information  Technology  Tools 
a .  Conclusions 

Weapon  system  programs  that  use  Information 
Technology  (IT)  tools  or  applications  to  complement  risk 
management  processes  can  effectively  manage  many  of  the 
risk  areas  discussed  in  this  thesis.  The  AAAV  program  used 
the  Virtual  Design  Database  (VDD)  during  PDRR  to  augment 
the  formal  risk  management  process. 

VDD  enabled  the  program  management  office  (PMO) 
to  identify,  categorize,  communicate  and  file  risks  through 
a  relational  database  available  on  the  program's  Intranet. 
Acknowledging  the  need  to  expand  VDD's  capabilities  to 
manage  risks  during  SDD,  the  AAAV  PMO  is  developing  Life 

Cycle  Information  System  (LCIS)  .  LCIS  expands  on  VDD  by 
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incorporating  applications  that  assist  in  tracking  risk 
trends  and  conducting  variance  analysis  to  assess 
mitigation  efforts:  a  capability  the  PMO  recognizes  as 
important  in  managing  risks  in  SDD .  LCIS,  a  web  based 

application,  will  improve  the  program' s  communications  by 
making  the  application  available  to  sub  contractors  and 
potentially  to  remote  test  locations,  as  well.  It  is  being 
developed  to  support  the  anticipated  future  needs  of  the 
AAAV  as  a  Program  Management  Office  of  Life  Cycle  Support 
(PMOLCS) . 

The  benefits  that  IT  applications  provide  PMOs 

are  numerous.  LCIS  will  expand  on  VDD' s  ability  to 

communicate  emerging  risks,  track  risk  mitigation 

activities  and  conduct  risk  analyses  to  support  program 
level  risk  management  efforts. 

b.  Recommendations 

Based  on  the  conclusions  discussed  above,  this 
thesis  offers  recommendations  for  managing  risk  in  weapon 
system  programs  through  the  use  of  IT  applications: 

•  Develop  and  employ  electronic  resources  to 

facilitate  and  complement  the  program' s  formal 
risk  management  methodology 

•  Make  this  IT  resource  available  to  all  program 

stakeholders:  Government,  Support  Contractors, 

Prime  and  Sub  Contractors 

•  Ensure  all  users  are  properly  trained 

•  Keep  the  application  simple 

•  The  application  should  support  the  program' s 

specific  goals  or  efforts  in  an  acquisition  phase 

•  Anticipate  desired  expansion  of  the  tool's 
capabilities  to  satisfy  program  requirements  in 
later  program  phases 
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•  When  practical,  employ  a  dedicated  program  Chief 

Information  Officer  (CIO)  to  oversee  IT 

initiatives 

2 .  The  Joint  Government -General  Dynamics  Risk 

Management  and  Resolution  Process 

a .  Conclusions 

A  weapon  system' s  risk  management  process  should 
be  simple  and  sustainable.  The  success  of  a  process  will 
depend  in  large  part  upon  the  degree  to  which  its 
implementation  does  not  detract  from  concurrent  program 
demands.  Risk  management  is  a  continuous  part  of  the 
system  development  and  acquisition  process.  All  activities 
in  the  process  should  add  measurable  value  to  the  program's 
development  and  production.  Risk  mitigation  activities 
need  to  be  tied  to  metrics  to  evaluate  progress  and 
efficacy  of  the  efforts. 

The  formal  risk  management  process  should  focus 
on  what  is  important  to  the  program:  meeting  user 

requirements  given  time  and  resource  constraints.  The 
process  should  involve  senior  leadership  participation 
while  maintaining  an  environment  that  encourages  the  airing 
and  resolution  of  risks.  It  should  reward  initiative  and 
acknowledge  the  value  of  responsible  risk  acceptance  while 
insisting  on  accountability  and  ownership  of  risk  and  its 
mitigation . 

b.  Recommendations 

The  analysis  and  conclusions  of  the  AAAV' s  risk 
management  and  resolution  process  drive  the  following 
recommendations : 
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•  The  risk  management  process  should  be  simple 

•  A  program  management  office  should  consider  using 
a  risk  primer  to  familiarize  and  train  personnel 
in  the  risk  management  process 

•  A  formal  risk  management  process  should  be 
sustainable  given  program  office  workforce 
strength  and  anticipated  demands 

•  Risk  management  activities  should  add  measurable 
value  to  the  program 

•  The  establishment  of  a  program  directorate  or 
division  to  oversee  risk  management  can  help  to 
coordinate  efforts,  track  risk  trends  and  liaise 
with  leadership 

•  Metrics  should  be  developed  and  employed  to 
assess  the  success  or  failure  of  mitigation 
efforts 

•  Program  offices  should  consider  periodic, 
external  risk  management  assessments 

3 .  Risk  Management  Through  the  Contracting  Process 
a .  Conclusions 

Contract  incentives  tied  to  a  Prime  contractor' s 
risk  management  process  are  an  effective  tool  to  transfer 
risk  from  the  Government  to  its  industry  counterpart.  The 
Government  program  office  ultimately  assumes  responsibility 
for  the  success  or  failure  of  a  system's  risk  handling. 
However,  tying  financial  incentives  for  the  contractor  to 
develop  and  implement  effective  risk  management  processes 
can  result  in  improved  contractor  performance.  GDAS'  award 
fee,  or  profit  sharing  structure  resulted  in  increased 
employee  buy-in  to  the  AAAV  risk  management  process  during 
PDRR.  As  a  result,  the  Government  continues  to  provide 
contractual  award  fee  incentives  for  GDAS  to  execute  a 
proactive  risk  management  process  in  SDD . 
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b.  Recommendations 

As  a  result  of  the  success  the  DRPM  AAA  has  had 
in  tying  risk  management  to  GDAS'  contract  award  fee  plan, 
this  thesis  offers  the  following  recommendations  concerning 
risk  management  through  the  contracting  process: 

•  Transferring  risk  from  the  Government  to 

contractors  through  financial  incentives  can  be 
an  effective  method  to  achieve  desired  results  or 
levels  of  effort 

•  Profit-sharing  organizational  structures  can 

incentivize  good  performance  and  employee  buy-in 

4.  Government  and  Prime  Contractor  Co-Location 
a .  Conclusions 

The  Marine  Corps/GDAS  co-location  on  the  AAAV 
program  has  created  an  environment  of  continual 

communication  between  Government  and  Prime  contractor 
personnel  across  traditional  program  lines.  The  ease  of 

communication  and  problem  resolution  decreases 
administrative  delay  times  and  miscommunications  between 
Government/contractor  counterparts.  Co -location  at  the 
point  of  vehicle  design,  testing,  assembly  and  prototype 
production  enhance  the  IPPD  process  by  providing  an 
environment  for  teams  to  truly  integrate. 

The  benefits  of  AAAV' s  co-location  are  evident  in 
the  proactive  management  of  risks  such  as  those  in 
Environmental  Safety  and  Health  (ESH) .  Co -location  allows 
Government  and  Industry  personnel  to  identify  and 
appreciate  different  organizational  cultures. 

Understanding  these  differences  enables  both  the  Marine 
Corps  AAAV  team  and  GDAS  to  work  towards  establishing  the 
most  productive  work  environment  for  the  benefit  of  the 
program. 
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b.  Recommendations 

Recommendations  concerning  Government  PMO  and 
Prime  contractor  co-location  are  as  follows: 

•  Co-location  facilitates  communication  and  the 
IPPD  process 

•  Co-located  PMOs  can  save  time  and  money  in  a 
system' s  developmental  stages 

•  User  representatives  involved  on  a  regular  basis 
in  a  system' s  design  for  suitability  can  prevent 
costly  and  avoidable  changes 

5 .  Test  and  Evaluation 

a .  Conclusions 

The  AAAV' s  test  program  in  PDRR  included 
combining  DT  and  OT  test  events  in  a  challenging 
operational  environment.  The  lessons  learned  from  the  test 
results  allowed  the  program  office  to  know  what  its 
strengths  and  deficiencies  were  entering  SDD .  This 

knowledge  shaped  the  SDD  test  plan  to  prepare  the  system  to 
meet  LRIP  entrance  criteria.  Testing  the  system 

aggressively  and  early  in  the  test  plan  reduced  the  risk  of 
being  forced  to  combine  risky  DT  and  OT  events  prior  to  a 
Milestone  decision  because  of  schedule  compression. 
Including  user  juries  and  Fleet  Marine  Force  (FMF)  Marines 
in  the  EOA  provided  the  program  with  relevant  user  feedback 
early  in  the  system' s  design  stages  when  changes  were 
possible  and  less  costly.  The  AAAV' s  emphasis  on  life 
cycle  support  testing  in  PDRR  represents  the  Marine  Corps' 
goal  of  reducing  total  ownership  cost  of  the  AAAV. 

b.  Recommendations 

Based  on  the  AAAV' s  test  history,  this  thesis 
offers  the  following  recommendations: 
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•  Conduct  logistics  and  life  cycle  support  tests 
with  user  juries  early  in  the  system' s 
development  to  avoid  costly  changes  and  reduce 
the  risk  of  fielding  an  unreliable  or  difficult 
to  maintain  system 

•  Combining  DT  and  OT  test  events  during  PDRR 
allows  a  program  to  refine  its  SDD  test  plan  to 
successfully  meet  LRIP  entrance  criteria 

•  Include  user  juries  in  DT  and  OT  test  events 
whenever  feasible  to  solicit  feedback  early  in 
the  system' s  development 

•  Ensure  all  personnel  involved  in  test  events  are 
thoroughly  trained  and  have  sufficient  experience 
to  be  able  to  execute  required  activities  at  an 
acceptable  level  of  performance. 

This  chapter  concludes  the  thesis  by  addressing 
what  conclusions  and  recommendations  can  be  made  from  the 
analysis  of  data  presented  in  previous  chapters.  The 

following  section  provides  suggested  areas  of  further 
research  in  risk  management  and  in  the  AAAV  program. 

C.  AREAS  OF  FURTHER  RESEARCH 

The  following  areas  of  further  research  are  suggested 
to  expand  upon  this  analysis  of  current  DoD  risk  management 
practices  in  the  development  and  procurement  of  complex 
weapon  systems: 

•  What  impact  do  the  revised  DoD  5000  Series 
acquisition  guidelines  have  on  DoD  developmental 
weapon  system  risk  management  practices? 

•  How  do  the  revised  DoD  5000  Series  acquisition 
guidelines  impact  the  AAAV  program  prior  to  and 
following  Milestone  C? 

•  What  conclusions  and  recommendations  can  be  made 
from  an  analysis  of  the  AAAV  program' s  SDD  risk 
management  strategy? 

•  What  conclusions  and  recommendations  can  be  drawn 
from  an  analysis  of  co-located  and  detached 
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program  management  offices  regarding  the  impact 
of  co-location  on  the  IPPD  process? 


What  risk  management  techniques  are  being  used  to 
manage  software  intensive  programs? 

What  metrics  can  be  used  to  evaluate  software 
development  and  are  they  effective? 
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APPENDIX.  LIST  OF  ACRONYMS 


AAAV 

AAV 

AoA 

APB 

ATD 

CAD 

CAIV 

CAN 

CARC 

CAX 

CHPPM 

CIO 

CPAF 

CSC 

DAB 

DAD 

DAU 

DoD 

E&MD 

EPRS 

DRPM 

DRPM  AAA 

DT&E 

ECP 

EOA 

EPMC 

ESH 

EV 


Advanced  Amphibious  Assault  Vehicle 
Amphibious  Assault  Vehicle 
Analysis  of  Alternatives 
Acquisition  Program  Baseline 
Advanced  Technology  Demonstration 
Computer  Aided  Design 
Cost  as  an  Independent  Variable 
Center  for  Naval  Analyses 
Chemical  Agent  Resistant  Coating 
Combined  Arms  Exercise 

Center  for  Health  Promotion  and  Preventive 
Medicine 

Chief  Information  Officer 
Cost  Plus  Award  Fee 
Computer  Systems  Corporation 
Defense  Acquisition  Board 
Defense  Acquisition  Deskbook 
Defense  Acquisition  University 
Department  of  Defense 

Engineering  and  Manufacturing  Development 
Electronic  Problem  Resolution  System 
Direct  Reporting  Program  Manager 

Direct  Reporting  Program  Manager  Advanced 
Amphibious  Assault 

Developmental  Test  and  Evaluation 
Engineering  Change  Proposal 
Early  Operational  Assessment 
Executive  Program  Managers  Course 
Environmental  Safety  and  Health 
Earned  Value 
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EVM 

EVMS 

FMF 

GAO 

GDAS 

GQM 

GSAM 

HAP 

IETM 

I-IPT 

I&A 

IOC 

IPPD 

IPT 

I  RAT 

IT 

ITP 

JTAV 

KPP 

LCAS 

LCC 

LCIS 

LRIP 

MCCDC 

MDA 

MNS 

M&S 

MS 

NEPA 

NMS 

NSS 


Earned  Value  Management 

Earned  Value  Management  System 

Fleet  Marine  Force 

Government  Accounting  Office 

General  Dynamics  Amphibious  Systems 

Goal,  Question,  Metric  [paradigm] 

Guidelines  for  Successful  Acquisition 
Software-Intensive  Systems 

Hazardous  Air  Pollutants 

Interactive  Electronic  Technical  Manual 

Integrating  Integrated  Product  Team 

Integration  and  Assembly 

Initial  Operational  Capability 

Integrated  Product  and  Process  Development 

Integrated  Product  Team 

Independent  Risk  Assessment  Team 

Information  Technology 

Integrated  Test  Program 

Joint  Total  Asset  Visibility 

Key  Performance  Parameter 

Landing  Craft  Air  Cushioned 

Lifecycle  Cost 

Life  Cycle  Information  System 

Low  Rate  Initial  Production 

Marine  Corps  Combat  Development  Command 

Milestone  Decision  Authority 

Mission  Needs  Statement 

Modeling  and  Simulation 

Milestone 

National  Environmental  Protection  Act 
National  Military  Strategy 
National  Security  Strategy 


OMFTS 

ODS 

ORD 

O&S 

OT&E 

PC 

PDRR 

P3I 

PM 

PMO 

PMOLCS 

PMT 

PP&I 

RCT 

RDT&E 

RFP 

RM 

RMC 

RMP 

RRB 

SDD 

SEI 

SOO 

SOW 

SSP 

TDP 

T&E 

TEMP 

TOC 

TPM 

USEPA 


Operational  Maneuver  From  the  Sea 
Ozone  Depleting  Substance 
Operational  Requirements  Document 
Operations  and  Support 
Operational  Test  and  Evaluation 
Personal  Computer 

Program  Definition  and  Risk  Reduction 
Pre-planned  Program  Improvement 
Program  Manager 
Program  Management  Office 

Program  Management  Office  of  Life  Cycle 
Support 

Program  Management  Team 
Program  Planning  and  Integration 
Risk  Coalition  Team 

Research,  Development,  Test  and  Evaluation 

Request  for  Proposal 

Risk  Management 

Risk  Management  Coordinator 

Risk  Management  Plan 

Risk  Resolution  Board 

System  Development  and  Demonstration 

Software  Engineering  Institute 

Statement  of  Objectives 

Statement  of  Work 

Source  Selection  Process 

Technical  Data  Package 

Test  and  Evaluation 

Test  and  Evaluation  Master  Plan 

Total  Ownership  Cost 

Technical  Performance  Measurement 

United  States  Environmental  Protection 
Agency 
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VDD 

VINTEGRA 

WATA 

WBS 

WG 

WSESRB 


Virtual  Design  Database 
Virtual  Integration  and  Assembly 
Worth  Avenue  Technology  Annex 
Work  Breakdown  Structure 
Working  Group 

Weapon  System  Explosive  Safety  Review  Board 
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